53-1003507-05

19 April 2016

Brocade Fabric OS
Extension
Administrator's Guide

Supporting Fabric OS v7.4.0 and Fabric OS 7.4.1

© 2016, Brocade Communications Systems, Inc. All Rights Reserved.

Brocade, Brocade Assurance, the B-wing symbol, ClearLink, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, VCS, VDX,
Vplane, and Vyatta are registered trademarks, and Fabric Vision is a trademark of Brocade Communications Systems, Inc., in the United
States and/or in other countries. Other brands, products, or service names mentioned may be trademarks of others.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any
equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document
at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be
currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in
this document may require an export license from the United States government.
The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the
accuracy of this document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that
accompany it.
The product described by this document may contain open source software covered by the GNU General Public License or other open
source license agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to
the open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.
Contents

Preface..................................................................................................................................... 7
Document conventions......................................................................................7
Text formatting conventions.................................................................. 7
Command syntax conventions.............................................................. 7
Notes, cautions, and warnings.............................................................. 8
Brocade resources............................................................................................ 9
Contacting Brocade Technical Support............................................................. 9
Document feedback........................................................................................ 10

About This Document.............................................................................................................. 11

Supported hardware and software.................................................................. 11
What's new in this document...........................................................................11

Brocade Extension Concepts and Features.............................................................................. 15

Brocade Extension concepts...........................................................................15
Brocade IP Extension concepts.......................................................... 15
VE_Ports and VEX_Ports................................................................... 16
Extension interfaces, circuits, and trunks............................................17
Extension Trunking .........................................................................................18
Redundancy and fault tolerance......................................................... 18
Tunnel restrictions for Fibre Channel Protocol and FICON
Acceleration...................................................................................19
IP WAN network considerations......................................................................19
Adaptive Rate Limiting ................................................................................... 20
Brocade 7840 Switch support for ARL................................................ 20
FSPF link cost calculation when ARL is used..................................... 21
Configuring ARL..................................................................................21
ARL configuration limitations...............................................................21
Compression options...................................................................................... 21
Brocade 7800 switch and FX8-24 blade............................................. 22
Brocade 7840 switch compression options.........................................22
FastWrite and Open Systems Tape Pipelining................................................ 24
FastWrite and OSTP configurations....................................................24
Support for IPv6 addressing............................................................................26
IPv6 with embedded IPv4 addresses..................................................27
Memory use limitations for large-device tunnel configurations....................... 27
Control blocks created during FCP traffic flow.................................... 28
Control blocks created during FICON traffic flow................................ 28
Effect of configuration on tunnel control block memory.......................29
Firmware downloads....................................................................................... 30

Extension Features on Brocade Extension Switches and Blades............................................... 33

Extension platforms and supported features...................................................33
Brocade 7800 Extension Switch..................................................................... 36
License options................................................................................... 36
Brocade FX8-24 Extension Blade................................................................... 37
Removing Brocade FX8-24 blades..................................................... 40

Brocade Fabric OS Extension Administrator's Guide 3

53-1003507-05
 License options................................................................................. 40
10 GbE port considerations...............................................................41
Multigigabit circuits............................................................................41
Crossports.........................................................................................41
Bandwidth allocation and restrictions................................................44
Brocade 7840 Extension Switch................................................................... 47
Brocade 7840 DP components and VE_Port distribution................. 48
10 GbE and 40 GbE port considerations.......................................... 51
Port grouping.....................................................................................51
10VE and 20VE port modes..............................................................52
Extension Hot Code Load................................................................. 53
IP Extension......................................................................................56
License options................................................................................. 59
Path Maximum Transmission Unit discovery................................................ 60
Tunnel and circuit requirements....................................................................61
General tunnel, circuit, and port requirements.................................. 61
Brocade 7800 extension switches.................................................... 61
Brocade FX8-24 extension blades.................................................... 63
Brocade 7840 extension switches.................................................... 64
Circuit failover............................................................................................... 65
Circuit Failover Grouping.................................................................. 66
10 GbE Lossless Link Loss (FX8-24 blade)......................................70
Failover in TI zones...........................................................................71
Bandwidth calculation during failover................................................72

Configuring Extension Features............................................................................................. 73

Configuration preparation............................................................................. 73
Configuration steps....................................................................................... 74
Setting VE_Ports to persistently disabled state............................................ 75
Disabling ports when FMS Mode is enabled.....................................75
Configuring VEX_Ports................................................................................. 76
Configuring the media type for GbE ports 0 and 1 (Brocade 7800 switch)... 76
Setting the GbE port operating mode on FX8-24 blade only........................ 76
Configuring switch and port modes (7840 switch)........................................ 77
Configuring port speed (Brocade 7840 switch)............................................. 79
Configuring an IPIF....................................................................................... 79
Configuring an IP route................................................................................. 80
Commands for configuring IP routes.................................................81
Commands for modifying IP routes...................................................82
Validating IP connectivity.............................................................................. 82
Creating an Extension tunnel........................................................................82
Creating an FX8-24 and Brocade 7800 tunnel................................. 83
Creating Brocade 7840 tunnels........................................................ 84
Tunnel configuration options............................................................. 84
Keep-alive timeout option..................................................................84
Creating additional circuits ...........................................................................85
Verifying the tunnel configuration..................................................................86
Configuring Extension HCL...........................................................................86
DP and tunnel configuration..............................................................87
Configuring backup tunnels...............................................................89
Enabling persistently disabled ports............................................................. 90
Disabling ports with FMS Mode enabled...........................................90
Modifying a tunnel.........................................................................................90
Modifying a circuit......................................................................................... 91
Deleting an IP interface ................................................................................91
Deleting an IP route ..................................................................................... 91
Deleting a trunk.............................................................................................92

4 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Deleting a circuit .............................................................................................92
Configuring Per-Priority TCP QoS priorities over a trunk................................92
Modifying default priority values......................................................................93
Using logical switches..................................................................................... 94
Logical switch overview ......................................................................94
Considerations for logical switches..................................................... 96
Managing QoS, DSCP, and VLANs...............................................................103
DSCP Quality of Service................................................................... 103
VLANs and Layer 2 Quality of Service..............................................103
Managing DSCP and VLAN support on circuits................................103
When both DSCP and L2CoS are used............................................104
Managing the VLAN tag table........................................................... 105
Implementing IPsec over tunnels.................................................................. 107
Limitations using IPsec over tunnels.................................................107
IPsec for the extension switches and blades.................................... 107
Enabling IPsec and IKE policies....................................................... 108
Brocade 7840 IKE authentication failures......................................... 109
Traffic Isolation Zoning.................................................................................. 110

Configuring IP Extension features.......................................................................................... 111

Configuration preparation for IP Extension features......................................111
Configuration steps for IP Extension features............................................... 112
Configuring hybrid mode for IP Extension features....................................... 113
Configuring GbE port for IP Extension LAN features.................................... 114
Configuring LAG............................................................................................ 115
Configuring switch virtual interface IPIF........................................................ 116
Configuring a tunnel to support IP Extension................................................ 117
Configuring bandwidth distribution................................................................ 118
Configuring tunnel compression.................................................................... 119
Configuring traffic control lists....................................................................... 120

Management and Troubleshooting........................................................................................ 123

In-band management.................................................................................... 123
IP routing...........................................................................................123
Configuring IP addresses and routes................................................124
VLAN tagging support....................................................................... 128
IP forwarding support........................................................................ 128
WAN performance analysis tools.................................................................. 129
The tperf option................................................................................. 130
Using ping to test a connection......................................................... 131
Using traceroute................................................................................132
Using WAN Tool................................................................................ 132
Using the portshow command.......................................................................137
Displaying IP interfaces.....................................................................137
Displaying IP routes.......................................................................... 137
Displaying switch mode information with the extncfg command....... 137
Displaying GbE port information with the portcfgge command......... 137
Listing the MAC addresses of LAN and GE ports............................. 137
Displaying LAG information...............................................................138
Displaying tunnel information............................................................ 138
Displaying tunnel HCL information....................................................139
Displaying TCL information............................................................... 139
Displaying IP Extension LAN statistics..............................................139
Displaying performance statistics......................................................140
Displaying QoS statistics...................................................................140
Displaying details.............................................................................. 140

Brocade Fabric OS Extension Administrator's Guide 5

53-1003507-05
 Displaying tunnel information (Brocade 7800)................................ 140
Displaying a tunnel with circuit information .................................... 140
Displaying tunnel performance .......................................................140
Displaying tunnel TCP statistics .....................................................141
Displaying circuits........................................................................... 141
Displaying a single circuit................................................................141
Displaying TCP statistics for circuits............................................... 141
Displaying circuit performance ....................................................... 141
Displaying QoS prioritization for a circuit........................................ 141
Displaying tunnel information (FX8-24 blade)................................. 142
Tunnel issues ............................................................................................. 142
Tunnel does not come online.......................................................... 142
Tunnel goes online and offline........................................................ 143
Troubleshooting Extension links................................................................. 143
Gathering additional information..................................................... 144
Using FTRACE............................................................................................144
FTRACE configuration.................................................................... 145
Changing configuration settings......................................................146
Displaying FTRACE status on a DP complex................................. 149

6 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
Preface

● Document conventions......................................................................................................7
● Brocade resources............................................................................................................ 9
● Contacting Brocade Technical Support............................................................................. 9
● Document feedback........................................................................................................ 10

Document conventions
The document conventions describe text formatting conventions, command syntax conventions, and
important notice formats used in Brocade technical documentation.

Text formatting conventions

Text formatting conventions such as boldface, italic, or Courier font may be used in the flow of the text
to highlight specific words or phrases.

Format Description
bold text Identifies command names
Identifies keywords and operands
Identifies the names of user-manipulated GUI elements
Identifies text to enter at the GUI

italic text Identifies emphasis

Identifies variables
Identifies document titles

Courier font Identifies CLI output

Identifies command syntax examples

Command syntax conventions

Bold and italic text identify command syntax components. Delimiters and operators define groupings of
parameters and their logical relationships.

Convention Description
bold text Identifies command names, keywords, and command options.
italic text Identifies a variable.
value In Fibre Channel products, a fixed value provided as input to a command
option is printed in plain text, for example, --show WWN.

Brocade Fabric OS Extension Administrator's Guide 7

53-1003507-05
Notes, cautions, and warnings

Convention Description

[] Syntax components displayed within square brackets are optional.

Default responses to system prompts are enclosed in square brackets.

{x|y|z} A choice of required parameters is enclosed in curly brackets separated by

vertical bars. You must select one of the options.
In Fibre Channel products, square brackets may be used instead for this
purpose.

x|y A vertical bar separates mutually exclusive elements.

<> Nonprinting characters, for example, passwords, are enclosed in angle

brackets.

... Repeat the previous element, for example, member[member...].

\ Indicates a “soft” line break in command examples. If a backslash separates

two lines of a command input, enter the entire command at the prompt without
the backslash.

Notes, cautions, and warnings

Notes, cautions, and warning statements may be used in this document. They are listed in the order of
increasing severity of potential hazards.

NOTE
A Note provides a tip, guidance, or advice, emphasizes important information, or provides a reference
to related information.

ATTENTION
An Attention statement indicates a stronger note, for example, to alert you when traffic might be
interrupted or the device might reboot.

CAUTION
A Caution statement alerts you to situations that can be potentially hazardous to you or cause
damage to hardware, firmware, software, or data.

DANGER
A Danger statement indicates conditions or situations that can be potentially lethal or
extremely hazardous to you. Safety labels are also attached directly to products to warn of
these conditions or situations.

8 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade resources

Brocade resources
Visit the Brocade website to locate related documentation for your product and additional Brocade
resources.

You can download additional publications supporting your product at www.brocade.com. Select the
Brocade Products tab to locate your product, then click the Brocade product name or image to open the
individual product page. The user manuals are available in the resources module at the bottom of the
page under the Documentation category.

To get up-to-the-minute information on Brocade products and resources, go to MyBrocade. You can
register at no cost to obtain a user ID and password.
Release notes are available on MyBrocade under Product Downloads.
White papers, online demonstrations, and data sheets are available through the Brocade website.

Contacting Brocade Technical Support

As a Brocade customer, you can contact Brocade Technical Support 24x7 online, by telephone, or by e-
mail. Brocade OEM customers contact their OEM/Solutions provider.

Brocade customers
For product support information and the latest information on contacting the Technical Assistance
Center, go to http://www.brocade.com/services-support/index.html.
If you have purchased Brocade product support directly from Brocade, use one of the following methods
to contact the Brocade Technical Assistance Center 24x7.

Online Telephone E-mail

Preferred method of contact for non- Required for Sev 1-Critical and Sev [email protected]
urgent issues: 2-High issues:
Please include:
• My Cases through MyBrocade • Continental US: 1-800-752-8061
• Problem summary
• Software downloads and licensing • Europe, Middle East, Africa, and
• Serial number
tools Asia Pacific: +800-AT FIBREE
(+800 28 34 27 33) • Installation details
• Knowledge Base
• For areas unable to access toll • Environment description
free number: +1-408-333-6061
• Toll-free numbers are available in
many countries.

Brocade OEM customers

If you have purchased Brocade product support from a Brocade OEM/Solution Provider, contact your
OEM/Solution Provider for all of your product support needs.
• OEM/Solution Providers are trained and certified by Brocade to support Brocade® products.
• Brocade provides backline support for issues that cannot be resolved by the OEM/Solution Provider.

Brocade Fabric OS Extension Administrator's Guide 9

53-1003507-05
Document feedback

• Brocade Supplemental Support augments your existing OEM support contract, providing direct
access to Brocade expertise. For more information, contact Brocade or your OEM.
• For questions regarding service levels and response times, contact your OEM/Solution Provider.

Document feedback
To send feedback and report errors in the documentation you can use the feedback form posted with
the document or you can e-mail the documentation team.
Quality is our first concern at Brocade and we have made every effort to ensure the accuracy and
completeness of this document. However, if you find an error or an omission, or you think that a topic
needs further development, we want to hear from you. You can provide feedback in two ways:
• Through the online feedback form in the HTML documents posted on www.brocade.com.
• By sending your feedback to [email protected].
Provide the publication title, part number, and as much detail as possible, including the topic heading
and page number if applicable, as well as your suggestions for improvement.

10 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
About This Document

● Supported hardware and software.................................................................................. 11

● What's new in this document...........................................................................................11

Supported hardware and software

The following hardware platforms support Brocade Extension (Fibre Channel over IP features and IP
Extension features) as described in this manual:
• Brocade DCX, DCX-4S, DCX 8510-4, and DCX 8510-8 with one or more FX8-24 blades
• Brocade 7800 switch
• Brocade 7840 switch

What's new in this document

This document includes new and modified information for the Fabric OS 7.4.0.

Major new additions or deletions in this document support the new IP Extension features related to the
Brocade 7840 Extension Switch, changes for Fabric OS v7.4.0, and corrections. An additional change
to this document is the repositioning of the Brocade Fibre Channel over IP (FCIP) features as the
Brocade Extension features. Note that Brocade Fabric OS (FOS) commands that support Extension still
use "fcip".
Major sections of this publication affected by additions and corrections include the following:
• Extension Concepts and Features
‐ Added Tunnel restrictions for Fibre Channel Protocol and FICON Acceleration on page 19
information for the IP Extension features.
‐ In the section IP WAN network considerations on page 19, clarified that the network must allow
ESP traffic to pass when using IPsec.
‐ In the section Memory use limitations for large-device tunnel configurations on page 27, updated
memory information for the Brocade 7840 from 512 MB to 1.3 GB.
‐ In the section Effect of configuration on tunnel control block memory on page 29, updated the
memory threshold information for FOS 7.4.0.
‐ In the section Firmware downloads on page 30, updated Extension HCL information for Fabric
OS 7.4.0 concurrent operation.
• Extension Features on Brocade Extension Switches and Blades
‐ In the section Redundancy and fault tolerance on page 18, added a note about using Lossless
Dynamic Load Sharing (DLS) with multiple parallel tunnels.
‐ In the section Brocade FX8-24 Extension Blade on page 37, and in the section Crossports on
page 41, corrected information about XGE port numbering supported by Brocade FX8-24 blade
and DP complexes.
‐ In the section Brocade 7840 switch considerations and limitations on page 102, restrictions are
removed for using the 7840 switch as a base switch.

Brocade Fabric OS Extension Administrator's Guide 11

53-1003507-05
About This Document

‐ Added IP Extension on page 56 to describe IP Extension features on the Brocade 7840 switch.
‐ In the section Removing Brocade FX8-24 blades on page 40, updated the instructions for blade
removal.
‐ In the section Front-end and back-end bandwidth on page 44, corrected the illustration showing
DP0 and DP1 port associations.
‐ In the section Multigigabit circuits on page 51, added information on maximum number of
circuits per tunnel.
‐ In the section Extension Hot Code Load on page 53, changed primary tunnel (PT) to main
tunnel (MT).
‐ In the section Extension HCL Limitations and considerations on page 55, added information
about using LLL when parallel tunnels are configured.
‐ Added sections describing IP Extension, beginning with IP Extension on page 56.
‐ In the section Examples of circuit failover in groups on page 67, a configuration example was
removed.
‐ In the section Configuring circuit failover groups on page 68, redundant information was
removed.
• Configuring Extension
‐ In the 7.3.0 release, this information was known as Configuring FCIP.
‐ In the section Configuring switch and port modes (7840 switch) on page 77, add information for
Brocade 7840 switch modes to include FCIP mode and hybrid mode.
‐ In the section Configuration steps on page 74, updated information for the Brocade 7840.
‐ In the section Setting the GbE port operating mode on FX8-24 blade only on page 76, updated
the output of the bladecfggemode --show command.
‐ In the sectionConfiguring switch and port modes (7840 switch) on page 77, updated
configuration modes.
‐ In the section Configuring port speed (Brocade 7840 switch) on page 79, updated the
configuration steps and removed an unsupported portCfgGe ge4 --set -speed auto command.
‐ In the section Commands for modifying IP routes on page 82, updated the information to
indicate the command is for the Brocade 7840 only.
‐ In the section Tunnel configuration options on page 84 , tables were removed describing tunnel
and circuit options for the portcfg fciptunnel create command and the portcfg fciptunnel
modify command. You can refer to Fabric OS Command Reference for information about the
portcfg fciptunnel create command and the portcfg fciptunnel modify command.
‐ The section "Creating a trunk (example)" showing an example of multi-circuit trunk creation has
been removed.
‐ In the section Configuring Extension HCL on page 86, replaced the terms primary tunnel and
PT with main tunnel and MT. The ouptut example was updated for FOS 7.4.0.
‐ In the section Connecting logical switches on page 96, added support for Brocade 7840 switch.
‐ In the section Port sharing example on page 97, updated the command output example.
‐ In the section Brocade 7800 switch considerations and limitations on page 101, updated the
allowed downgrade version from v7.0.0 to v7.1.0.
‐ In the section Enabling XISL for VE_Ports (FX8-24 blade / 7840 switch) on page 102, removed
the limits on using Brocade 7800 and FX8-24 as base switches for XISL. XISL for Brocade 7840
was added.
‐ In the section Managing DSCP and VLAN support on circuits on page 103, removed the table of
command options for the portcfg fciptunnel create command and the portcfg fciptunnel
modify command.
‐ In the section VLAN tagging examples on page 104, added a note about setting the VLAN ID for
the Brocade 7840 switch.
‐ In the section Limitations using IPsec over tunnels on page 107, updated IPsec information for
the Brocade 7840 switch.
‐ In the section IPsec for the extension switches and blades on page 107, updated information
about the SA lifetime.

12 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 About This Document

‐ In the section Enabling IPsec and IKE policies on page 108, added information for the Brocade
7840 IKE key, and updated the example.
‐ Added the section Brocade 7840 IKE authentication failures on page 109.
• Configuring IP Extension
‐ Added the chapter Configuring IP Extension for the IP Extension feature configuration tasks. All
content is new for 7.4.0.
• Extension Management and Troubleshooting
‐ In the section Redundant connections to the management stations example on page 126,
updated the configuration example.
‐ In the section Using the portshow command on page 137, added the following new information for
IP Extension on the Brocade 7840.
‐ Displaying switch mode information with the extncfg command on page 137
‐ Displaying GbE port information with the portcfgge command on page 137
‐ Displaying LAG information on page 138
‐ Displaying tunnel HCL information on page 139
‐ Displaying TCL information on page 139
‐ Displaying IP Extension LAN statistics on page 139
‐ Listing the MAC addresses of LAN and GE ports on page 137
‐ Updated the section Gathering additional information on page 144 to include commands useful for
IP Extension features.

Changes made for this release (53-1003507-02)

Minor changes have been made to the document to enhance presentation of the graphics and
illustrations.

Changes made for this release (53-1003507-03)

In the section IP WAN network considerations on page 19, added information about not dropping the
URG flag from ports 3225 and 3226.

Changes made for this release (53-1003507-04)

In Firmware downloads on page 30 and Extension HCL Limitations and considerations on page 55,
added information that when Teradata emulation is configured on a Brocade 7840 switch, Extension
HCL is not supported.

Changes made for this release (53-1003507-05)

In Brocade 7800 Extension Switch on page 36, corrected information to state total throughput is 4
Gbps.
In Keep-alive timeout option on page 84, added information that the default keep-alive timeout values
(KATOV) for the portcfg fciptunnel command will differ depending on platform and FICON emulation
status.
In Configuration steps for IP Extension features on page 112, added a note that IP Extension end
points must be on different subnets.

Brocade Fabric OS Extension Administrator's Guide 13

53-1003507-05
What's new in this document

14 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
Brocade Extension Concepts and Features

● Brocade Extension concepts...........................................................................................15

● Extension Trunking .........................................................................................................18
● IP WAN network considerations......................................................................................19
● Adaptive Rate Limiting ................................................................................................... 20
● Compression options...................................................................................................... 21
● FastWrite and Open Systems Tape Pipelining................................................................ 24
● Support for IPv6 addressing............................................................................................26
● Memory use limitations for large-device tunnel configurations....................................... 27
● Firmware downloads....................................................................................................... 30

Brocade Extension concepts

Brocade Extension products support FC/FICON based data flows as well as IP based storage data
flows.

Brocade Extension enables you to use the existing IP wide are network (WAN) infrastructure to connect
Fibre Channel and IP fabrics. Brocade Extension supports applications such as remote data replication
(RDR), centralized backup, and data migration over very long distances that are impractical or very
costly using native Fibre Channel or IP connections. Extension tunnels, built on a physical connection
between two extension switches or blades, allow Fibre Channel and IP I/O to pass through the IP WAN.
The extension tunnel and TCP connections ensure in-order delivery of Fibre Channel (FC) and IP
frames and lossless transmission. The Fibre Channel fabric and all targets and initiators, whether FC or
IP, are unaware of the presence of the IP WAN.

Brocade IP Extension concepts

The Brocade 7840 Extension switch supports IP based storage data flows as well as FC/FICON based
data flows. IP extensions provide enterprise-class support for IP storage applications, using existing IP
wide area network (WAN) infrastructure. IP Extension features are offered only on the Brocade 7840
Extension Switch platform.
IP data flows across Brocade tunnels are referred to as IP Extension. IP Extension provides storage
administrators with the means to monitor and manage their IP storage flows across an IP infrastructure.
IP Extension enables you to use the existing IP WAN infrastructure to connect IP storage. Additionally,
IP Extension gives you visibility and control of flows using various diagnostic tools, IPsec, compression,
QoS, Extension Trunking, and lossless tunnel resiliency. IP Extension supports applications such as
array native IP remote data replication (RDR), IP based centralized backup, VM replication; host based
and database replication over IP, NAS head replication between data centers, data migration between
data centers and others.
Brocade WAN Optimized TCP (WO-TCP) ensures in-order lossless transmission of IP Extension data.
IP Extension establishes a proxy TCP endpoint for local devices. Local devices are unaware and
unaffected by the latency and quality of the IP WAN. This accelerates end device native TCP. IP

Brocade Fabric OS Extension Administrator's Guide 15

53-1003507-05
VE_Ports and VEX_Ports

Extension data across the IP WAN uses WO-TCP, a highly efficient and aggressive TCP stack for
moving data between data centers.
IP Extension provides the following advantages:
• Data Center Interconnect (DCI) — Unified support and management of both FC/FICON and IP
• Storage Administrators — Provision once and over time connect many devices
• High performance for high speed WAN links (one or more 10 Gbps and 40 Gbps links)
• WAN bandwidth pooling — pool bandwidth from multiple links/providers
• Lossless link loss (LLL)
• Adaptive Rate Limiting (ARL)
• Network resiliency and high availability using Extension Trunking
• Efficient protocol transport — negligible added overhead
• TCP Acceleration with WAN Optimized TCP
• Streams — Virtual windows on WAN Optimized TCP to eliminate head of line blocking (HoLB)
• High speed compression using Deflate
• High speed IPsec (AES 256)
• Diagnostic and troubleshooting tools — WAN Health and Wtool
• Separate QoS for both FCIP and IP Extension with DSCP and/or 802.1P marking and enforcement
• 9216 byte Jumbo Frames for both LAN and WAN networks

FIGURE 1 Extension tunnel concept and TCP/IP layers for FCIP and IP Extension

VE_Ports and VEX_Ports

Any FC communications between switches that include fabric services needs to communicate across
an ISL using E_Ports. There are various types of E_Ports:
• E_Port
• EX_Port
• VE_Port
• VEX_Port
VE and VEX ports are virtual because they are extension tunnel facing. These are the ports that
enable communication across an extension tunnel. EX and VEX ports are FC Routed ports. Router
ports are the demarcation point of fabric services for a fabric. Fabric services do not extend beyond an
EX or VEX port.

16 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Extension interfaces, circuits, and trunks

E port types No FCE FCR

Native FC E_Port EX_Port

Extended over tunnel VE_Port VEX_Port

NOTE
VEX_Ports are not supported on the Brocade 7840 switch.
Once the tunnels are configured and the WO-TCP connections are made for a circuit, a logical
interswitch link (ISL) is established between the switches. VE_Ports operate like E_Ports for all fabric
services and Fabric OS operations, except that VE_Ports use TCP/IP and Ethernet as the transport
instead of FC.
A "virtual" EX_Port exposed by the extension tunnel to form an ISL connection allows you to configure a
virtual EX_Port or VEX_Port to support FCR demarcation. From the point of view of a switch in an edge
fabric, a VEX_Port appears as a normal E_Port. It follows the same Fibre Channel protocol as other
E_Ports. However, VEX_Ports terminate the attached fabric at the port and do not allow fabrics to
merge by propagating fabric services or routing topology information beyond that edge fabric. This
provides edge fabric,or remote edge fabric isolation outward from the EX_Port or VEX_Port.

NOTE
VE_Ports or VEX_Ports cannot connect in parallel to the same domain at the same time as Fibre
Channel E_Ports or EX_Ports.
An extension tunnel is assigned to a VE_Port or VEX_Port on the switch or blade at each end of the
tunnel. Because multiple VE_Ports and VEX_Ports can exist on the extension switch or blade, you can
create multiple tunnels.
Fibre Channel frames enter an extension tunnel through virtual E_Ports (VE_Ports) or virtual extension
ports (VEX_Ports) and are encapsulated and passed to TCP layer connections. A Data Processing
(DP) complex on the switch or blade handles the FC frame encapsulation, de-encapsulation, and
transmission to the TCP link.

Extension interfaces, circuits, and trunks

A circuit is a connection between a pair of IP addresses that are associated with source and destination
endpoints of an extension tunnel. Circuits provide the links for traffic flow between source and
destination interfaces that are located on either end of the tunnel. For each tunnel, you can configure a
single circuit or a trunk consisting of multiple circuits. Multiple circuits can be configured per Ethernet
port by assigning them unique IP interfaces (IPIFs). When you configure a circuit, you provide the IP
addresses for its source and destination interfaces.
You must configure unique IPIFs at the endpoint of each circuit. An IPIF consists of an IP address for a
circuit endpoint, subnet mask, and an MTU size. For the Brocade 7840 switch, the IPIF can also contain
a VLAN ID for VLAN tagging and the ability to enable PMTU. If the remote IPIF is not on the same
subnet as the local IPIF, you must configure an IP route to that destination specifying the gateway to
use. You can define a specific number of routes per IPIF based on the extension platform. Refer to
Tunnel and circuit requirements on page 61 for specifications. An Ethernet interface can contain
multiple IPIFs.

NOTE
In this publication, the "source" or "local" is the switch you are configuring, while the "destination" or
"remote" is the switch on the other end of the tunnel.

Brocade Fabric OS Extension Administrator's Guide 17

53-1003507-05
Extension Trunking

Configure an extension tunnel by specifying a VE_Port for a source and destination interface. When
you configure a circuit on the tunnel, you will provide two IP addresses, one for the source and one for
the destination IP interface.
For Extension Hot Code Load (HCL) tunnels, four IP addresses are configured per circuit, which
includes both endpoints. The four addresses are the local and remote IP addresses, and the local and
remote HA IP addresses used by HCL. In most instances, the two local IP addresses are in the same
subnet and the two remote IP addresses are in the same subnet. All IP addresses must be able to
communicate across the IP infrastructure. Extension HCL is supported only on the Brocade 7840. For
additional information, refer to Configuring Extension HCL on page 86.

FIGURE 2 Extension tunnel and circuits

An extension trunk is a tunnel consisting of multiple circuits.

For specifications and restrictions on tunnels, circuits, and trunks for the Brocade 7800 Extension
Switch, 7840 Extension Switch, and the FX8-24 Extension Blade, refer to the Extension Features on
Brocade Extension Switches and Blades chapter.

Extension Trunking
Extension Trunking is a method for managing the use of WAN bandwidth and providing redundant
paths over the WAN that can protect against transmission loss due to WAN failure. Extension Trunking
also provides granular load balancing on a weighted round-robin basis per batch. Trunking is enabled
by creating multiple circuits within a tunnel so that the tunnel utilizes multiple circuits to carry traffic
between multiple source and destination addresses. For circuit capacities for Brocade extension
switches and blades, refer to Tunnel and circuit requirements on page 61.

Redundancy and fault tolerance

Multiple extension tunnels can be defined between pairs of extension switches or blades, but doing so
defeats the benefits of a multiple-circuit extension tunnel. Defining two tunnels between a pair of
switches or blades is not as redundant or fault-tolerant as having multiple circuits in one tunnel.
Extension Trunking provides lossless link loss (LLL). LLL ensures all data lost in flight is retransmitted
and placed back in order prior to being delivered to upper layer protocols. This is an essential feature
to prevent interface control checks (IFCCs) on mainframes using FICON and SCSI timeouts for open-
system-based replication. For more information about LLL on specific Brocade extension switches and
blades, refer to Circuit failover on page 65.

18 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Tunnel restrictions for Fibre Channel Protocol and FICON Acceleration

NOTE
When you create multiple parallel tunnels between the same switch domains, you must enable Lossless
Dynamic Load Sharing (DLS). This is because there can be routing updates that will occur when
tunnels come up or go down. Each routing update can cause dropped, or unrouteable frames if the
destination is via the peer tunnel connected switch domain.

Tunnel restrictions for Fibre Channel Protocol and FICON Acceleration

Multiple extension tunnels within the same switch are not supported between pairs of extension
switches or blades when protocol optimization features—FastWrite, OSTP, and FICON Acceleration—
are enabled on the tunnel, unless Traffic Isolation (TI) zones or logical switch/logical fabric (LS/LF)
configurations are used to provide deterministic flows between the switches. These features require
deterministic FC frame routing between initiators and targets when multiple tunnels or VE_Ports exist.
Non-controlled, parallel (equal-cost multi-path) tunnels are not supported between domains when
protocol optimization is enabled on one or more tunnels without controlling the routing of SID/DID pairs
to a specific tunnel using TI Zones or Virtual Fabrics (VF) LS/LF configurations.
Note the following additional restrictions:
• FICON networks with or without FICON Acceleration do not support exchange-based routing (EBR)
configurations.
• The recommended best practice is to have identical FOS versions at both ends of an extension
tunnel.
• When planning Fabric OS upgrades or downgrades, it is recommended that you upgrade or
downgrade both endpoints of an extension tunnel with the same FOS version. When using Extension
HCL in FOS 7.4.0, the firmware upgrade process is done concurrently.
• When configuring tunnels to support large numbers of devices, consider memory limitations of the
Brocade extension switch or blade if you are enabling any type of protocol optimization feature. If too
many devices are present or activated at one time, protocol optimization such as FICON Acceleration
can be negatively impacted. Refer to Memory use limitations for large-device tunnel configurations
on page 27.

IP WAN network considerations

Because Brocade Extension tunnels use TCP connections over an existing wide area network, consult
with the WAN carrier and IP network administrator to ensure that the network hardware and software
equipment operating in the data path can properly support the TCP connections. Keep the following
considerations in mind:
• Routers and firewalls that are in the data path must be configured to pass traffic through a specific
TCP port on the switch. If IPsec is used, the network must allow Encapsulating Security Payload
(ESP) traffic to pass through (UDP port 500). The Brocade WO-TCP implementation selects a port
between 49152 and 65535 as the ephemeral (or initiating) port to open up to port 3225 and 3226.
• On the Brocade 7840 switch, the TCP URG flag is frequently set. This flag indicates the beginning of
a tunnel header in the TCP packet. Make sure these flags are not dropped from ports 3225 and
3226.
• The Brocade 7800 switch and Brocade FX8-24 blade use TCP port 3225.
• The Brocade 7840 switch uses ports 3225 and 3226.

Brocade Fabric OS Extension Administrator's Guide 19

53-1003507-05
Adaptive Rate Limiting

• To enable recovery from a WAN failure or outage, be sure that diverse, redundant network paths
are available across the WAN.
• Be sure the underlying WAN infrastructure can support the redundancy and performance expected
in your implementation.

Adaptive Rate Limiting

Adaptive Rate Limiting (ARL) is performed on circuits to change the rate in which the tunnel transmits
data through the IP network. ARL uses information from the TCP connections to determine and adjust
the rate limit for the circuit dynamically. This allows connections to utilize the maximum available
bandwidth while providing a minimum bandwidth guarantee. ARL is configured on a per-circuit basis
because each circuit may have available different amounts of bandwidth.
ARL is supported only if Fabric OS v7.0.0 and later is running on both ends of the tunnel. For Fabric
OS v7.0.0 and later, you can configure minimum and maximum rates for each circuit of a tunnel using
the following ports:
• XGE (10 GbE) ports on the Brocade FX8-24 blade (xge0 and xge1).
• 10G (1/10 GbE) ports on the Brocade 7840 switch (ge2-ge17).
• 40 GbE ports on the Brocade 7840 switch (ge0 and ge1).
• 1 GbE ports on the Brocade 7800 switch (ge0-ge5) and Brocade FX8-24 blade (ge0-ge9).
ARL applies a minimum and maximum traffic rate, and allows the traffic demand and WAN connection
quality to determine the rate dynamically. If traffic is flowing error-free over the WAN, the rate grows
towards the maximum rate. If TCP reports an increase in retransmissions, the rate reduces towards
the minimum. ARL never attempts to exceed the maximum configured value and reserves at least the
minimum configured value.
For ARL limitations and features specific to supported products, refer to the Extension Features on
Brocade Extension Switches and Blades on page 33 chapter.

Brocade 7840 Switch support for ARL

ARL on the Brocade 7840 has been enhanced to react ten times faster to varying traffic patterns that
compete for WAN bandwidth or use shared interfaces.
ARL has always accommodated shared bandwidth; however, the amount of storage data using
Extension connections continues to grow and consume larger and faster links. On the Brocade 7840,
the enhanced response time of ARL provides faster rate limiting adaptation, which permits optimized
throughput of not only Extension traffic, but also the competing flows.
The back-off mechanism implemented by ARL is optimized to increase overall throughput. ARL
dynamically preserves bandwidth and evaluates network conditions to see whether additional back-
offs are required.
ARL maintains Round Trip Time (RTT) stateful information to better predict network conditions and to
allow intelligent and granular decisions about proper adaptive rate limiting. When ARL encounters a
network error, it looks back at prior stateful information, which will be different relative to the current
state. Rate limit decisions are then modified using the ARL algorithm. When configured for automatic
selection, ARL will dynamically determine which algorithm to use based on the changing network
conditions.

20 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 FSPF link cost calculation when ARL is used

FSPF link cost calculation when ARL is used

Fabric Shortest Path First (FSPF) is a link state path selection protocol that directs traffic along the
shortest path between the source and destination based upon the link cost. When ARL is used, the link
cost is equal to the sum of the maximum traffic rates of all established, currently active low metric
circuits in the tunnel. The following formulas are used:
• If the bandwidth is greater than or equal to 2 Gbps, the link cost is 500.
• If the bandwidth is less than 2 Gbps, but greater than or equal to 1 Gbps, the link cost is 1,000,000
divided by the bandwidth in Mbps.
• If the bandwidth is less than 1 Gbps, the link cost is 2000 minus the bandwidth in Mbps.
When running multiple parallel tunnels, set the static link costs if you want all tunnels used at all times.
Otherwise, a circuit outage can take a tunnel out of service because of a link cost change. If multiple
parallel tunnels are used, configure Lossless DLS. This avoids circuit bounce because of disruptive
bandwidth updates.

Configuring ARL
To configure the minimum and maximum committed rates for ARL on a circuit, refer to Tunnel
configuration options on page 84.

ARL configuration limitations

Consider the following limitations when configuring ARL:
• As a best practice, the aggregate of the circuit minimum rate bandwidth settings through a VE_Port
tunnel should not exceed the bandwidth of the WAN link. For example, if the WAN link is 500 Mbps,
the aggregate of the ARL maximum rates connected to that WAN link can be no more than 500
Mbps. For ingress rates, there is no limit because the FC flow control (BBC) rate limits the incoming
data.
• The aggregate of the minimum configured values cannot exceed the speed of the Ethernet interface,
which is 1 Gbps for GbE ports or 10 Gbps for 10 GbE ports, or 40 Gbps for 40 GbE ports.
• Configure minimum rates of all the tunnels so that the combined rate does not exceed the
specifications listed for the extension product in the Tunnel and circuit requirements on page 61.
• For 1 GbE, 10 GbE, and 40 GbE ports, the ratio between the minimum committed rate and the
maximum committed rate for a single circuit cannot exceed five times the minimum. For example, if
the minimum is set to 2 Gbps, the maximum for that circuit cannot exceed 10 Gbps. This is enforced
in software.
• The ratio between any two circuits on the same tunnel should not exceed four times the lower circuit.
For example, if one circuit is configured to 1 Gbps, any other circuit in that same tunnel should not
exceed 4 Gbps. Except for the Brocade 7840, this is not enforced in software, but is strongly
recommended.

Compression options
Compression options are defined in the portcfg fciptunnel port create and portcfg fciptunnel port
modify commands. There are different options for different extension products.

Brocade Fabric OS Extension Administrator's Guide 21

53-1003507-05
Brocade 7800 switch and FX8-24 blade

NOTE
Throughput for all compression modes depends on the compression ratio achievable for the data
pattern. Brocade makes no promises, guarantees, or assumptions about compression ratio that any
application may achieve.

Brocade 7800 switch and FX8-24 blade

The following compression options are available for the Brocade 7800 switch and FX8-24 blade:
• Standard - A hardware compression mode.
• Moderate - A combination of hardware and software compression that provides more compression
than hardware compression alone. This option supports up to 8 Gbps of FC traffic.
• Aggressive - Software-only compression that provides a more aggressive algorithm than used for
the standard and moderate options. This option supports up to 2.5 Gbps of FC traffic.
• Auto - Allows the system to set the best compression mode based on the tunnel’s configured
bandwidth and the aggregate bandwidth of all tunnels in the extension switch or blade.
Follow the guidelines for assigning explicit compression levels for tunnels in the following table.

TABLE 1 Assigning compression levels

Total effective tunnels FC side Compression level

Equal to or less than 512 Mbps Aggressive

More than 512 Mbps and less than or equal to 2 Gbps Moderate

More than 2 Gbps Standard

Brocade 7840 switch compression options

The following compression options are available for the Brocade 7840 switch.
• None - no compression.
• Deflate - Processor-based compression. This option initiates the processor compression engine in
deflate mode with preference on speed. This provides a lower speed than fast-deflate, but more
than aggressive deflate (16 Gbps total per DP before compression). The compression is better than
fast deflate, but is typically not as good as aggressive deflate.

22 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade Extension Concepts and Features

• Aggressive deflate - Processor-based compression. Initiates the processor engine in deflate mode
with preference on compression. This is the slowest (10 Gbps before compression), but typically
provides the best compression.
• Fast deflate - Hardware-based compression. This option initiates a deflate-based algorithm to
compress data before it enters the DP and decompresses the data after it leaves the DP (egress).
This is the highest throughput mode of compression (40 Gbps per DP before compression), but
provides the least amount of compression.
Follow the guidelines for assigning explicit compression levels for tunnels in the following table.

TABLE 2 Assigning compression levels

Total tunnel bandwidth on a DP Compression level

2 Gbps or less Aggressive deflate

2 Gbps to 4 Gbps Deflate

More than 4 Gbps Fast deflate

The enhancements for IP Extension allow you to configure compression on the tunnel at a protocol
level. The compression options override the main tunnel compression level and set the compression for
the specified protocol to the desired mode. The available modes depend on the protocol, whether FC or
IP.

TABLE 3 IP Extension hybrid mode protocol compression choices

Compression level FC protocol support IP protocol support

Deflate Yes Yes

Fast deflate Yes No

Agressive deflate Yes Yes

Brocade Fabric OS Extension Administrator's Guide 23

53-1003507-05
FastWrite and Open Systems Tape Pipelining

FastWrite and Open Systems Tape Pipelining

Brocade FastWrite is an algorithm that reduces the number of round trips required to complete a SCSI
write operation. FastWrite can maintain throughput levels over links that have significant latency. The
RDR (Remote Data Replication) application still experiences latency; however, reduced throughput
due to that latency is minimized for asynchronous applications, and response time is cut in half for
synchronous applications.
Open Systems Tape Pipelining (OSTP) can be used to enhance open systems SCSI tape read and
write I/O performance. When the Extension link is the part of the network with the longest latency,
OSTP can provide accelerated speeds for tape read and write I/O over tunnels. To use OSTP, you
must enable FastWrite also.
OSTP accelerates SCSI read and write inputs and outputs to sequential devices (such as tape drives)
over Extension, which reduces the number of round-trip times needed to complete the I/O over the IP
network and speeds up the process.
Both sides of a tunnel must have matching configurations for OSTP and FastWrite features to work.
FastWrite and OSTP are enabled by turning them on during the tunnel configuration process. They are
enabled on a per-tunnel basis.

FastWrite and OSTP configurations

The FCP features used in FastWrite and OSTP require a single deterministic path between initiators
and targets when multiple tunnels exist. If there are non-controlled parallel (ECMP) tunnels between
the same SID/DID pairs, protocol optimization will fail when a command is routed over one tunnel and
the response is returned over a different tunnel. To help understand the supported configurations,
consider the configurations shown in the following two figures. In both cases, there are no multiple
ECMP paths. In the first figure, there is a single tunnel with FastWrite and OSTP enabled. In the
second figure, there are multiple tunnels, but none of them create a multiple ECMP path.

NOTE
Only one emulating tunnel is supported between an initiator port and a peer device port.
Brocade extension devices have the intelligence to distinguish between storage flows that use protocol
optimization and those that do not use protocol optimization. For example, IBM SVC does not use
FastWrite, but EMC SRDF/A does use FastWrite. Both applications functioning over the connection
are fully supported for FastWrite because FastWrite will not engage with the IBM SVC flows while still
engaging with the SRDF/A flows across the same VE_Port. This is also true when using OSTP with
IBM SVC. Both flows can utilize the same VE_Port with FastWrite and OSTP enabled. The IBM SVC
will not engage the protocol optimization.

24 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade Extension Concepts and Features

FIGURE 3 Single tunnel, FastWrite and OSTP enabled

FIGURE 4 Multiple tunnels to multiple ports, FastWrite and OSTP enabled on a per-tunnel, per-port
basis

In some cases, Traffic Isolation Zoning or VF LS/LF configurations may be used to control the routing of
SID/DID pairs to individual tunnels. This provides deterministic flows between the switches and allows
the use of ECMP. Refer to the Fabric OS Administrator’s Guide for more information about Traffic
Isolation Zoning.

Brocade Fabric OS Extension Administrator's Guide 25

53-1003507-05
Support for IPv6 addressing

Support for IPv6 addressing

The IPv6 implementation is a dual-IP layer operation implementation as described in RFC 4213. IPv6
addresses can exist with IPv4 addresses on the same interface, but the circuits must be configured as
IPv6-to-IPv6 and IPv4-to-IPv4 connections. IPv6-to-IPv4 connections are not supported. Likewise,
encapsulation of IPv4 in IPv6 and IPv6 in IPv4 is not supported.
This implementation of IPv6 uses unicast addresses for the interfaces with circuits. Unicast addresses
must follow the RFC 4291 IPv6 standard. This IPv6 implementation uses the IANA-assigned IPv6
Global Unicast address space (2000::/3). The starting three bits must be 001 (binary) unless IPv6 with
embedded IPv4 addresses is used. The link-local unicast address is automatically configured on the
interface, but using the link-local address space for circuit endpoints is not allowed. Site-local unicast
addresses are not allowed as circuit endpoints.
Note the following IPv6 addressing points:
• Anycast addresses are not used. Each IPv6 interface has a unique unicast address and addresses
configured are assumed to be unicast.
• Multicast addresses cannot be configured for an IPv6 interface with circuits. The IPv6 interface
does not belong to any Multicast groups other than the All-Nodes Multicast and the Solicited-Node
Multicast groups (these do not require user configuration).
• The IPv6 implementation follows the RFC 2460 standard for the 40-byte IPv6 header format.
• The IPv6 8-bit Traffic class field is defined by the configured Differentiated Services field for IPv6
(RFC 2474). The configuration of this is done on the circuit using the Differentiated Services Code
Point (DSCP) parameters to fill the 6-bit DSCP field.
• Flow labels are not supported on this IPv6 implementation. The 20-bit Flow Label field defaults to all
zeros.
• The IPv6 optional Extension Headers are not supported. The optional Extension Headers (zeros)
inserted into any ingress packets that contain these headers will be discarded. The next header
field must be the Layer 4 protocol for this implementation.
• Parts of the Neighbor Discovery protocol (RFC 4861) are used in this implementation.
‐ Hop limits (such as Time to Live (TTL)) are learned from the Neighbor Advertisement packet.
‐ The link-local addresses of neighbors are learned from Neighbor Advertisement.
‐ The netmask is deprecated in IPv6. Instead, the prefix length notation is used to denote subnets
in IPv6 (the Classless Inter-Domain Routing (CIDR) addressing syntax). Prefix length of neighbor
nodes is learned from the received Neighbor Advertisement packet.
‐ The IPv6 link-local address for each GE interface is configured at startup and advertised to
neighbors. The user does not configure the interface link-local address.
• The 8-bit hop limit field is filled by the learned value during Neighbor Discovery.
• IPv6 addresses and routes must be statically configured by the user. Router Advertisements and
IPv6 Stateless Address Autoconfiguration (RFC 2462) are not supported.
• The Neighbor Discovery ICMPv6 Solicitations and Advertisements are transmitted to the Layer 2
Ethernet multicast MAC address derived from the IPv6 source address (RFC 2464).
• ICMPv6 message types in RFC 4443 and ICMPv6 message types used for Neighbor Discovery are
supported.
• Path MTU Discovery
‐ For the Brocade 7800 switch and FX8-24 blade, Path MTU (PMTU) discovery is not supported.
The MTU option in the portcfg ipif command is optional. If not configured, 1500 bytes is used.
The maximum IP MTU supported is 1500 bytes (including the 40-byte fixed IPv6 header), the
same as for IPv4. The minimum IP MTU allowed is 1280 bytes, including the 40-byte fixed IPv6
header. Any network used for IPv6 circuits must support an IP MTU of 1280 bytes or larger. IPv6

26 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 IPv6 with embedded IPv4 addresses

fragmentation is not supported. The Layer 4 protocol ensures that the PDU is less than the IP
MTU (including headers).
‐ For the Brocade 7840 switch, PMTU discovery is supported. Refer to Path Maximum Transmission
Unit discovery on page 60.
• IPv6 addressing with IPsec:
‐ For the Brocade 7840 switch, IPv6 addressing can be used when implementing IPsec.
‐ For the Brocade 7800 switch and FX8-24 blade, IPv6 addressing cannot be used when
implementing IPsec.

IPv6 with embedded IPv4 addresses

Only IPv4-compatible IPv6 addresses are supported. Only the low-order 32 bits of the address can be
used as an IPv4 address (the high-order 96 bits must be all zeros). This allows IPv6 addresses to be
used on an IPv4 routing infrastructure that supports IPv6 tunneling over the network. Both endpoints of
the circuit must be configured with IPv4-compatible IPv6 addresses. IPv4-to-IPv6 connections are not
supported. IPv4-mapped IPv6 addresses are not supported because they are intended for nodes that
support IPv4 only when mapped to an IPv6 node.

Memory use limitations for large-device tunnel configurations

The data processing layer on the Brocade extension switch and blade data processing (DP) complex
has access to reserved memory used for control block structure allocations. Following are related
specifications for the Brocade switches and blades.

TABLE 4 VE_Ports and DRAM pool sizes for extension products

Product DP VE_Ports DP DRAM pool size

Brocade 7800 DP0 - 16 through 23 200 MB

Brocade 7840 DP0 - 24 through 33 1.3 GB

DP1 - 34 through 43 1.3 GB

Brocade FX8-24 DP0 - 22 through 31 268 MB

DP1 - 12 through 21 268 MB

Use the portshow xtun slot/ve -dram2 command to display current consumption of the tunnel DP
complex control block memory pool. Following is an example of command use with the portion of output
showing total DRAM2 pool size and current consumption for a Brocade 7840 switch.

switch:admin> portshow xtun 25 -dram2

Dram2 Pool Info:

Brocade Fabric OS Extension Administrator's Guide 27

53-1003507-05
Control blocks created during FCP traffic flow

---------------------------------------------
Total Bytes in DRAM2 Pool: 1336910592 (free) 1504640 (fastfreed)
Total DRAM Bytes Allocated: 6771328 (in use)

Tunnel processing will create more control blocks when any type of emulation feature is enabled, such
as FCP or FICON. In those cases, be sure to not include too many devices running over the tunnel. If
too many devices are present or activated at one time, emulation operations can be negatively
impacted. Even without emulation enabled, too many devices running over the tunnel may impact
operations at some point because of memory consumption. Note that a configuration that works
without an emulation feature, such as FICON Acceleration, FastWrite, or Open Systems Tape
Pipelining (OSTP), may not work when emulation features are enabled.

Control blocks created during FCP traffic flow

For FCP traffic flows, tunnel processing creates control block structures based upon the SID/DID pairs
used between initiators and devices. If either FastWrite or OSTP (read or write) is enabled, additional
structures and control blocks are created for each logical unit number (LUN) on a SID/DID-pair basis.
FCP processing in an emulated tunnel configuration will create multiple control blocks for each LUN if
there are multiple SID/DID pairs that can be used to access that same LUN. Each FCP-identified
SID/DID flow will be recorded in a structure called an ITN (initiator, target, nexus). Each specific LUN
on a SID/DID flow will have an ITL (initiator, target, LUN) control block created for the flow. FCP
emulation processing also creates a structure for each outstanding FC exchange called a turbo write
block (TWB).

Control blocks created during FICON traffic flow

For FICON traffic flows, tunnel processing creates a control block structure based upon the SID/DID
pairs called a FICON device path block (FDPB). If any FICON emulation feature is enabled, additional
control blocks are created for each SID/DID pair, logical partition (LPAR) number (FICON channel
block structure), LCU number (FICON control unit block structure), and for each individual FICON
device address on those LCUs (FICON device control block structure).
The total number of FICON device control blocks that will be created over a FICON emulating tunnel is
represented by the following equation:
FDCBs = Host Ports x Device Ports x LPARs x LCUs x FICON Devices per LCU
This number grows quickly in extended direct-attached storage device (DASD) configurations, such as
those used in IBM z/OS Global Mirror/XRC configurations.

FDCBs example
Assume that the tunnel is used to extend two channel paths (CHPIDs) from a System Data Mover
(SDM) site to a production site. Assume also that there are two SDM-extended LPARs and that the
IBM DS8000 production controllers have 32 LCUs per chassis, and each LCU has 256 device
addresses.
Using the preceding equation, the number of extended FICON device control block images created
would be the following:
2 Host Ports * 2 Device Ports * 2 LPARs * 32 LCUs * 256 Devices per LCU = 56,536 FICON device
control block images

28 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Effect of configuration on tunnel control block memory

Effect of configuration on tunnel control block memory

The portshow xtun slot/ve-port -fcp -port -stats command displays current usage and control block
sizes per tunnel once control blocks have been allocated. Use output from this command to determine
the unique characteristics for a specific tunnel configuration. The highlighted text in the following
example shows statistics for the control block structures created for FCP and FICON traffic flows during
tunnel processing.

portshow xtun 1/12 -fcp -port -stats

Slot(1) VePort(12) Port Stats: OK
Global Queue Stats:
Name,cnt,max,usage,size,total size
Data,12,14,44,8192,98304
Message,0,1,9,1432,0
Stat,40,40,80,928,37120
Stat Cache,0,40,40,0,0
Global stats,0,0,0,0,153240
Port Queue Stats:
Name,cnt,max,usage,size,total size
Image,40,40,80,0,0
SRB,0,0,0,0,0
TWB,1,40,356,0,0
Port Struct Allocation Stats:
Name,cnt,max,usage,size
IMAGE,40,40,80,3560
ITN,26,0,0,2984
ITL,1103,0,0,2312
FDPB,30,0,0,3864
FCHB,108,0,0,1400
FCUB,1721,0,0,1592
FDCB,14738,0,0,920
Global Buffer Stats:
Name,current,min,max
Write Data Storage,0,0,0
Read Data Storage,0,0,0
XBAR % avail,98,98,100
WIRE % avail,95,95,100

Use output from portshow xtun slot/ve-port -fcp -port -stats command in conjunction with output from
the portshow xtun slot/ve-port -dram2 command to determine how a tunnel configuration is affecting
tunnel control block memory. As a rule of thumb, no more than 80 percent of the tunnel DP complex
control block memory pool (dram2) should be allocated for SID/DID pair-related control blocks (ITNs,
ITLs, FDPBs, FCHBs, FCUBs, and FDCBs). When more than 80 percent of the pool is allocated, the
tunnel configuration should be redesigned to ensure continuous operation. The design should include
examining the existing number of SID/DID pairs in the configuration and determining whether new
switches, chassis, or blades should be acquired to reduce the percentage of current usage of the
DRAM2.
For Fabric OS v7.2.0 and later, RASlog message XTUN-1008 provides notification of DRAM2 memory
usage. The message is generated by the DP complex when significant memory thresholds are reached.
The following thresholds are shown for the Brocade 7800 switch and FX8-24 blade.
For Fabric OS 7.4.0 and later, each FX8-24 blade, 7800 and 7840 DP complex generates the
XTUN-1008 RASLOG message when the following percentages of the DRAM memory pool are
available:
• 50%
• 25%
• 12.5%
• 6.25%
• .05%
The RASlog message contents include the amount of allocated memory from the pool, the amount of
free memory in the pool, and the total pool size. Use the RASlog message contents to determine if you
need to reduce the size of the extended configuration or to plan for additional switch resources.

Brocade Fabric OS Extension Administrator's Guide 29

53-1003507-05
Firmware downloads

Brocade switches and blade DPs are expected to support no more than the number of FICON device
control blocks (FDCBs) and extended LUNs (ITLs) noted in the following table.

TABLE 5 FDCBs and ITLs per product DP

Product FDCB ITL

Brocade 7800 120,000 50,000

Brocade 7840 512,000 200,000

Brocade FX8-24 160,000 65,000

The Brocade 7840 switch has 1.3 GB of DRAM2 memory allocated per DP. During hot code load
(HCL) operations, duplicated emulation control blocks are created on the same DP for the high-
availability portion of the tunnel. That means that at one point in time during the Extension HCL
process, twice the normal memory requirements are consumed. This duplication process occurs on
the remote non-Extension HCL DP when the primary local DP is going through feature disable
processing.
The amount of DRAM2 memory on the Brocade 7840 should be able to support Extension HCL
operations with approximately 512K FICON devices active through the VE_Ports on that DP.
Because each customer configuration is unique, the supported number and types of devices will be
different. In large configurations, the administrator should review memory usage periodically to ensure
continued, reliable operations of the tunnel and emulation features.

Firmware downloads
For the Brocade 7800 switch and FX8-24 blade, if Fibre Channel traffic or FCIP traffic is active on
Fibre Channel ports, the traffic will be disrupted during a firmware download.
The Brocade 7840 switch supports the Extension hot code load (HCL) feature. During an Extension
HCL action, traffic is failed over to one DP complex as firmware upgrades in the other DP complex.
With Extension HCL, active FC traffic on Fibre Channel ports and VE_Ports is not disrupted during a
firmware download. For more information on this process, refer to Extension Hot Code Load on page
53.

NOTE
When the Brocade 7840 is operating in hybrid mode, Extension HCL is disruptive to IP traffic on a
VE_Port. IP traffic will be disrupted during a firmware download.

NOTE
When Teradata Emulation is enabled on an Extension tunnel, Extension HCL is not supported. You
must perform a disruptive firmware download.

30 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade Extension Concepts and Features

The best practice is to update the switch or blade at both ends of the tunnel with the same maintenance
release of Fabric OS.
For details on downloading firmware, refer to the chapter on installing and maintaining firmware in the
Fabric OS Administrator’s Guide.

Brocade Fabric OS Extension Administrator's Guide 31

53-1003507-05
Firmware downloads

32 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
Extension Features on Brocade Extension Switches and Blades

● Extension platforms and supported features...................................................................33

● Brocade 7800 Extension Switch..................................................................................... 36
● Brocade FX8-24 Extension Blade................................................................................... 37
● Brocade 7840 Extension Switch..................................................................................... 47
● Path Maximum Transmission Unit discovery.................................................................. 60
● Tunnel and circuit requirements...................................................................................... 61
● Circuit failover................................................................................................................. 65

Extension platforms and supported features

There are three Brocade platforms that support Extension features:

• The Brocade 7800 switch
• The Brocade FX8-24 blade (Brocade DCX, DCX-4S, DCX 8510-8, and DCX 8510-4 chassis)
• The Brocade 7840 switch
Note the following about extension connections between these products:
• Extension connections are not supported between the Brocade 7800 switch or FX8-24 blades and
previous generation products such as Brocade 7500 switches or FR4-18i blades.
• Extension connections are not supported between Brocade 7840 and previous generation products,
such as Brocade 7800 switches, Brocade 7500 switches, FX8-24 blades, and FR4-18i blades.
The following table provides details about platform capabilities.

TABLE 6 Extension capabilities by platform

Capabilities Brocade 7800 Brocade FX8-24 Brocade 7840

Extension Trunking Yes Yes Yes

Adaptive Rate Limiting Yes Yes Yes

10 GbE ports No Yes Yes (1/10 Gbps)

Brocade Fabric OS Extension Administrator's Guide 33

53-1003507-05
Extension Features on Brocade Extension Switches and Blades

TABLE 6 Extension capabilities by platform (Continued)

Capabilities Brocade 7800 Brocade FX8-24 Brocade 7840

40 GbE ports No No Yes

Enabled using 7840 WAN

Rate Upgrade 2 license.

FC ports Yes (1, 2, 4, 8 Gbps) Yes (1, 2, 4, 8 Gbps) Yes (2, 4, 8, 16 Gbps)

Compression Yes Yes Yes

LZ (Lempel-Ziv) and LZ and Deflate Deflate, Aggressive

Deflate Deflate, Fast Deflate

Protocol acceleration Yes Yes Yes

• Brocade Fastwrite
• Open Systems Tape
Pipelining

‐ OSTP read
‐ OSTP write

QoS Yes Yes Yes

• Marking DSCP
• Marking 802.1P - VLAN
tagging
• Enforcement 802.1P -
VLAN tagging

FICON extension Yes Yes Yes

• FICON emulation
• IBM z/OS Global Mirror
(formerly eXtended
Remote Copy or XRC)
acceleration
• Tape read acceleration
• Tape write acceleration
• Teradata emulation
• Printer emulation

34 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Extension Features on Brocade Extension Switches and Blades

TABLE 6 Extension capabilities by platform (Continued)

Capabilities Brocade 7800 Brocade FX8-24 Brocade 7840

IPsec Yes Yes Yes

• AES-256-GCM Transport mode encrypted Transport mode encrypted Transport mode encrypted
• SHA-512 HMAC data transfer (ESP) method data transfer (ESP) method data transfer (ESP) method
• IKEv2

VEX_Ports Yes Yes No

Support for third-party Yes Yes1 No

WAN optimization
Support limited to Silver
hardware
Peak for Fabric OS v7.1.0b
and later and to Riverbed
for Fabric OS v6.4.x.

IPv6 addresses for Yes2 Yes2 Yes

extension tunnels

Support for jumbo frames No No Yes

IP MTU of 1500 is IP MTU of 1500 is IP MTU of 9216 is

maximum maximum maximum

Path Maximum No No Yes

Transmission Unit (PMTU)
Maximum discoverable
Discovery
size is 9100 bytes.

IP Extension No No Yes

• Hybrid mode IP traffic through a Brocade

• Link access group extension tunnel.
(LAG)
Deflate and aggressive
• Switch virtual interface deflate IP compression
(SVI) IPIF options are supported, but
• IP compression not fast deflate.
• Traffic control list (TCL)
• LAN side jumbo frames

The following notes apply to the preceding table:

1. Not supported in Fabric OS v7.0 and later.
2. IPv6 addressing is not supported in conjunction with IPsec.

Brocade Fabric OS Extension Administrator's Guide 35

53-1003507-05
Brocade 7800 Extension Switch

Brocade 7800 Extension Switch

This section provides information on ports, circuits, and tunnels specific to the Brocade 7800
Extension Switch.
The following figure shows the FC ports and GbE ports on the Brocade 7800 switch. There are 16 FC
ports, numbered 0 through 15. The FC ports can operate at 1, 2, 4, or 8 Gbps. There are six GbE
ports, numbered 0 through 5. Ports 0 and 1 are available as either RJ-45 ports or small form-factor
pluggable (SFP) transceiver ports. Only six total GbE ports can be used. The six GbE ports together
can provide up to 4 Gbps total bandwidth (full duplex).

FIGURE 5 Brocade 7800 switch FC and GbE ports

1. FC ports 0 through 3
2. FC ports 4 through 15
3. Copper GbE ports 0 and 1 (These ports are RJ-45 copper alternatives for GbE ports 0 and 1.)
4. GbE ports 0 through 5
The Brocade 7800 switch comes in two models:
• The Brocade 7800 4/2 base model uses FC ports 0 through 3, and GbE ports 0 and 1. The GbE
ports can be either copper or optical. The RJ-45 copper ports are the default ports. Consider the
following when using these ports:
‐ Copper ports do not support auto-sense functions.
‐ With copper media, auto-negotiation must be enabled on the other end of the port connection.
• The Brocade 7800 16/6 uses FC ports 0 through 15, and GbE ports 0 through 5. The 7800 Upgrade
license is required. A 7800 Upgrade license can be purchased for a Brocade 7800 4/2, which
enables 12 more Fibre Channel ports for a total of 16, and enables the use of 4 more optical GbE
ports for a total of 6.

License options
Important extension capabilities of the Brocade 7800 switch require feature licenses. Use the
licenseshow command to display license keys and licenses currently installed.

NOTE
FCR is not supported on a Brocade 7800 switch that has been partitioned for Virtual Fabrics.

36 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade FX8-24 Extension Blade

TABLE 7 Brocade 7800 feature licenses

Feature Purpose License (licenseShow

output)

7800 upgrade • Enables full hardware capabilities on the Brocade 7800 base 7800 Upgrade license
switch, increasing the number of Fibre Channel ports from
four to sixteen and the number of GbE ports from two to six.
• Supports up to eight extension tunnels instead of two.
• Supports advanced capabilities such as Open Systems Tape
Pipelining (OSTP).

NOTE
You must reboot the switch to activate this license.

Advanced FICON Enables accelerated tape read/write and IBM z/OS Global Advanced FICON
acceleration Mirror, Teradata, and printer emulation features in FICON Acceleration (FTR_AFA)
environments. Slot-based license. license

Integrated routing Required to configure VEX_Ports to support Fibre Channel Integrated Routing license
(IR) Routing (FCR). Chassis-based license.

Advanced extension Required for multiple-circuit tunnels, Trunking, Adaptive Rate Advanced Extension
Limiting (ARL), and other Extension features. (FTR_AE) license

For complete information about the licenses described in the preceding table and additional licenses
available for the switch, refer to the Fabric OS Software Licensing Guide.

Brocade FX8-24 Extension Blade

This section provides information on ports, circuits, and tunnels specific to the Brocade FX8-24
Extension Blade.
The figure below shows the FC ports, GbE ports, and 10 GbE ports on the Brocade FX8-24. There are
12 FC ports, numbered 0 through 11. The FC ports can operate at 1, 2, 4, or 8 Gbps. There are ten
GbE ports, numbered 0 through 9. Ports xge0 and xge1 are 10 GbE ports.
The Brocade FX8-24 supports two FCIP data processor (DP) complexes. Each DP complex has a
home, or local, 10 GbE XGE interface and controls a specific range of GbE and VE_Ports.
The DP0 complex has the following ports:
• VE_Ports 22 through 31
• XGE port 0

Brocade Fabric OS Extension Administrator's Guide 37

53-1003507-05
Extension Features on Brocade Extension Switches and Blades

• Has a maximum bandwidth of 10 Gbps

• Operates in 10 Gbps mode and dual mode
The DP1 complex has the following ports:
• VE_Ports 12 through 21
• GbE ports 0 through 9 while operating in 1 Gbps mode and dual mode
• XGE port 1 while operating in 10Gbps mode
• Has a maximum bandwidth of 10 Gbps
The Brocade FX8-24 allows a maximum of 20 Gbps full-duplex bandwidth for tunnel connections and
can operate in one of three different modes:
• 1 Gbps mode—You can use all ten GbE ports (0 through 9). Both XGE ports are disabled.
• 10 Gbps mode—You can use the xge0 and xge1 ports. GbE ports (0 through 9) are disabled.
• Dual mode—You can use GbE ports 0 through 9, and port xge0. The xge1 port is disabled.
The Brocade FX8-24 can be deployed in a Brocade DCX, DCX-4S, DCX 8510-8, or DCX 8510XT-4
chassis.
The following figure illustrates ports and LED indicators on the Brocade FX8-24.

38 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Extension Features on Brocade Extension Switches and Blades

FIGURE 6 Brocade FX8-24 FC and GbE ports

1. Power LED
2. GbE ports 0 through 3
3. 10 GbE ports (Labeled xge0 and xge 1 on the blade.)
4. FC ports 0 through 5
5. Status LED
6. GbE ports 4 through 9
7. FC ports 6 through 11

Brocade Fabric OS Extension Administrator's Guide 39

53-1003507-05
Removing Brocade FX8-24 blades

Removing Brocade FX8-24 blades

ATTENTION
If you are permanently removing a blade from a Brocade DCX, DCX-4S, DCX 8510-8, or DCX 8510-4
chassis to relocate to another slot in the chassis or you are removing the blade from the chassis
entirely, you must follow these procedures before removing the blade.

1. Delete all fciptunnel configurations via the portcfg fciptunnel slot / vePort command.
2. Delete all IP Routes defined on the blade to be removed via the portcfg iproute command.
3. Delete all IPIFs defined on the blade via the portcfg ipif slot/geX | xgeX command .
4. If logical switches are used on the switch, move all FX8-24 ports back to the default logical switch
5. Remove the slot from the chassis

License options
Important extension capabilities of the FX8-24 blade require the feature licenses shown in the
following table. Use the licenseShow command to display license keys and licenses currently
installed.

TABLE 8 Brocade FX8-24 feature licenses

Feature Purpose License (licenseShow output)

10 GbE support Allows 10 Gbps operation on 10 GbE ports. Slot- 10 Gigabit FCIP/Fibre Channel
based license. (FTR_10G) license

Advanced FICON Enables accelerated tape read/write and IBM z/OS Advanced FICON Acceleration
acceleration Global Mirror, Teradata, and printer emulation (FTR_AFA) license
features in FICON environments. Slot-based license.

Integrated routing (IR) Required to configure VEX_Ports to support Fibre Integrated Routing license
Channel Routing (FCR). Chassis-based license.

Advanced extension Required for multiple-circuit tunnels, Trunking, Advanced Extension (FTR_AE)
Adaptive Rate Limiting (ARL), and other features. license
Slot-based license.

For complete information about the licenses described in the preceding table and additional licenses
available for the switch, refer to the Fabric OS Software Licensing Guide.

40 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 10 GbE port considerations

10 GbE port considerations

Enhanced 10 GbE port operation is different than GbE port operation and requires special
considerations when configuring circuits, tunnels, failover operations, and bandwidth. For a complete list
of tunnel, circuit, and IP address requirements and capacities, refer to Brocade FX8-24 Extension Blade
on page 37.

Multigigabit circuits
For each 10 GbE port on an FX8-24 blade, you can configure multigigabit circuits. For example, a single
10 Gbps circuit or two 5 Gbps circuits can be configured per port. A limit of ten circuits can be
configured on a single port. The blade at each end of the tunnel must be running Fabric OS v7.0 or later
if the committed rate for circuits exceeds 1 Gbps. The maximum committed rate for a circuit between 10
GbE ports is 10 Gbps.

NOTE
There is no difference in latency or throughput performance for single or multigigabit circuits.

Crossports
When a DP complex is not using its local XGE interface (xge0 or xge1), but is using the alternate or
remote interface, that interface is known as a crossport. The crossport for xge0 is xge1 and for xge1,
the crossport is xge0. Crossports are supported only in the FX8-24 blade.
Typically, IP interface addresses (IPIFs) used by ge0 through ge9 and xge1 are used for any circuits
that use VE_Ports 12 through 21. The xge1 port is the local XGE interface for VE_Ports 12 through 21.
Likewise, IP addresses configured for xge0 are used by circuits for VE_Ports 22 through 31.
Configure a crossport by assigning an IP address to the remote XGE port that can be used by the local
XGE port. For example, assigning an IP address to xge0 as a crossport makes the address available on
the remote xge0 for VE_Ports 12 through 21 on the local xge1.
You can also assign IP routes (iproutes) used by the local port, VLAN tagging, and circuits with metrics
to the remote XGE port to allow failover to the crossports.
Crossports contain the IP interface addresses (IPIFs) and IP routes (iproutes) that belong to the remote
interface. To use crossports, both XGE ports must be configured in 10 Gbps mode.

Configuring crossports
Configure crossport XGE port addresses using the --crossport or -x (shorthand) options for the portcfg
ipif command, as shown in the following example. Note that in this example, IP address 192.168.11.20
is made available on xge0 for circuits on VE_Ports 12 through 21 on local port xge1.
1. Configure an interface for the local XGE port (xge1).
switch:admin> portcfg ipif 8/xge1 create 192.168.10.20 netmask
255.255.255.0 mtu 1500
Operation Succeeded
2. Configure interface 192.168.11.20 on remote port xge0 to be available for VE_Ports 12 through 21.
switch:admin> portcfg ipif 8/xge0 create 192.168.11.20 netmask 255.255.255.0
mtu 1500 --crossport

Brocade Fabric OS Extension Administrator's Guide 41

53-1003507-05
Configuring 10 GbE lossless failover with crossports

or

switch:admin> portcfg ipif 8/xge0 create 192.168.11.20 netmask 255.255.255.0

mtu 1500 -x

The output from portshow ipif for xge1 shows the crossport tag.
switch43:root>portshow ipif 8/xge1
Port IP Address / Pfx MTU VLAN Flags
-------------------------------------------------------------
8/xge1 192.168.10.20 / 24 1500 0 U R M
8/xge1 192.168.11.20 / 24 1500 0 U R M X

Delete the crossport address using the delete option instead of the create option for the portcfg ipif
command.

switch43:root>portcfg ipif 8/xge1 delete 192.168.11.20 netmask 255.255.255.0

mtu 1500 -x

When deleted, output from portshow ipif for xge1 will not show the crossport.
switch43:root>portshow ipif 8/xge1
Port IP Address / Pfx MTU VLAN Flags
-------------------------------------------------------------
8/xge1 192.168.10.20 / 24 1500 0 U R M

NOTE
If the crossport or -x option is not specified and the address is on the crossport, the command will
fail with an unknown IP address. The command will also fail if the crossport option is specified and
the address is not on the crossport.
Display local and crossport interface configuration details for a specific XGE port using the
portshow ipif slot/xgeport command. Use the portshow ipif command to display details for all
interfaces.
portshow ipif 8/xge0
portshow ipif

Configuring 10 GbE lossless failover with crossports

Refer to 10 GbE Lossless Link Loss (FX8-24 blade) on page 70.

Configuring IP routes with crossports

You can configure IP routes with crossport addresses using the portcfg iproute [slot\port] create
command, as in the following example. In the example, the route will be available for tunnel circuits
using VE ports 12 through 21.

portcfg iproute 8/xge0 create 1.1.1.0 netmask 255.255.255.0 192.168.11.250 --

crossport

or

portcfg iproute 8/xge0 create 1.1.1.0 netmask 255.255.255.0 192.168.11.250 -x

Delete the route using the delete option instead of the create option for the portcfg iproute
command.

portcfg iproute 8/xge0 delete 1.1.1.0 netmask 255.255.255.0 -x

42 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring VLAN tags with crossports

NOTE
If the crossport or -x option is not specified and the address is on the crossport, the command will fail
with an unknown IP address. The command will also fail if the crossport option is specified and the
address is not on the crossport.
Display the static IP routes for the local interface and crossport using the portshow iproute command:
portshow iproute 1/xge0

Display the IP interface configured for the local interface and crossport using the portshow ipif
command.

portshow ipif 1/xge0

For more information on configuring an IP route, refer to Configuring an IP route on page 80.

NOTE
If an XGE port has both regular and crossport addresses configured on it, and they use the same IP
route, then two routes must be configured: a regular route and an identical route on the crossport.

Configuring VLAN tags with crossports

Add entries with crossport addresses to the VLAN tag table using the portcfg vlantag [slot/port] add
command, as in the following example. This example allows VE ports 12 through 21 to use the
configured local IP interface with this VLAN tag.

portcfg vlantag 8/xge0 add 192.168.11.20 200 1 --crossport

or

portcfg vlantag 8/xge0 add 192.168.11.20 200 1 -x

Delete the VLAN tag using the delete option instead of the add option for the portcfg vlantag
command.

portcfg vlantag 8/xge0 delete 192.168.11.20 200 1 -x

Display VLAN tag configuration using the portshow vlantag command.

NOTE
To tag Class F traffic or data path traffic, use the -v or - -vlan-tagging option for the fcipcircuit create
or fcipcircuit modify command. The portcfg vlantag command is primarily used for ping and
traceroute operation and not for tunnels and circuits.
For more information on managing VLAN tags, refer to Managing the VLAN tag table on page 105.

Displaying VLAN tag configuration using the portshow vlantag command

Following is an example for displaying VLAN tagging information for port 0 on blade 8.

portshow vlantag 8/xge0

For more information on managing VLAN tags, refer to Managing the VLAN tag table on page 105.

Brocade Fabric OS Extension Administrator's Guide 43

53-1003507-05
Using ping with crossports

For more information on using Fabric OS commands, optional arguments, and command output refer
to the Fabric OS Command Reference.

Using ping with crossports

You can ping crossport addresses, as in the following example. Note that if the crossport or x options
are not specified and the address is on the crossport, the portCmd command will fail with an unknown
IP address.

portcmd --ping 8/xge0 -s 192.168.11.20 -d 1.1.1.1 --crossport

or

portcmd --ping 8/xge0 -s 192.168.11.20 -d 1.1.1.1 -x

When using VLANS, VLAN tagging ensures that test traffic traverses the same path as real traffic. A
VLAN tag entry for both the local and remote sides of the route must exist prior to using the portCmd
--ping command. Refer to Managing the VLAN tag table on page 105 for details.
For more information on using ping, refer to Using ping to test a connection on page 131.

Using traceroute with crossports

You can trace a route to a crossport address, as in the following example. Note that if the crossport or
x options are not specified and the address is on the crossport, the portCmd command will fail with an
unknown IP address. The command will also fail if the -x option is specified and the address is not on
the crossport.

portcmd --traceroute 8/xge0 -s 192.168.11.20 -d 1.1.1.1 --crossport

or

portcmd --traceroute 8/xge0 -s 192.168.11.20 -d 1.1.1.1 -x

When using VLANS, VLAN tagging ensures that test traffic traverses the same path as real traffic. A
VLAN tag entry for both the local and remote sides of the route must exist prior to using the portCmd
--traceroute command. Refer to Managing the VLAN tag table on page 105 for details.
For more information on using traceroute, refer to Using traceroute on page 132.

Bandwidth allocation and restrictions

There are specific bandwidth allocations and restrictions for the FX8-24 blade that are important to
review when configuring tunnels and circuits.

Front-end and back-end bandwidth

The FX8-24 blade contains internal port complex with 1 Gbps ports to support the blade’s VE_Port
groups, Data Processor (DP) complexes, GbE ports, and XGE ports.
Each DP complex has 10 Gbps (full-duplex) available bandwidth. Therefore, each VE_Port group
(VE_Port 22-31 and VE_Port 12-21) has 10 Gbps of bandwidth available to the internal port complex
back-end ports. When the tunnels using VE_Ports in a specific VE_Port group consume the group’s
back-end bandwidth, additional circuits cannot be created for those tunnels. The port complex has 10
Gbps of front-end bandwidth available for each of the XGE ports. Tunnels (VE_Ports) cannot consume
more than 10 Gbps bandwith over an XGE port. The internal port complex has another 10 Gbps of
bandwidth available for the crossport.

44 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Calculating back-end bandwidth

The following figure illustrates the internal DP complex with VE_Port groups, internal port complex,
front-end and back-end port areas, and the crossport (xport) on an FCX8-24 blade.

FIGURE 7 Internal port and DP complexes on FX8-24 blade

Calculating back-end bandwidth

The following are the ways for configuring the back-end bandwidth for tunnel and DP complex:
• To calculate the consumed bandwidth for a tunnel, round the maximum committed rates for all metric
0 circuits up to the next whole rate (for example 1.5 Gbps becomes 2 Gbps) and add them up. Then
add the rounded-up maximum committed rates for all metric 1 circuits. The greater of the two values
is the consumed bandwidth for a tunnel.
• To calculate the total consumed back-end port bandwidth for a DP complex, add the consumed
bandwidth for each tunnel in the DP complex VE_Port group. The total cannot exceed 10 Gbps.
• Back-end bandwidths are always rounded up for each VE_Port group. For example, a circuit defined
as 1.5 Gbps will consume 2 Gbps of back-end bandwidth.

Calculating front-end bandwidth

The following are the ways to calculate the front-end bandwidth for a tunnel and a XGE port:
• To calculate the front-end bandwidth usage on a per-tunnel and per-XGE port basis, add the
consumed bandwidth for all metric 0 circuits for a tunnel using xge0 or xge1. Add the total consumed
bandwidth for all metric 1 circuits for the tunnel. The greater of the two values is the total front-end
port bandwidth usage for an xge0 or xge1 tunnel. Refer to Circuit failover on page 65 for more
information on assigning metrics to circuits.
• Each XGE port is allocated 10 Gbps of front-end bandwidth. The total consumed front-end port
bandwidth cannot exceed 10 Gbps per XGE port.

Brocade Fabric OS Extension Administrator's Guide 45

53-1003507-05
Calculating crossport bandwidth

Calculating crossport bandwidth

The DP complexes share only one crossport, so total available bandwidth is 10 Gbps for all VE_Ports
on the blade, regardless of the DP complex to which the VE_Ports belong. For more information on
crossports, refer to Crossports on page 41.
• To calculate the consumed crossport bandwidth on a per-tunnel basis, add the consumed
bandwidth for all metric 0 circuits in the tunnel that use the crossport. Add the total consumed
bandwidth for all metric 1 circuits in the tunnel that use the crossport. The greater of the two values
is the total crossport-consumed bandwidth for the tunnel .
• The total crossport-consumed bandwidth is the total of the bandwidth for the tunnels using
VE_Ports 12 through 31. The total crossport-consumed bandwidth cannot exceed 10 Gbps.

ARL limits
Bandwidth allocations are subject to the minimum committed rate (-b) and maximum committed rate (-
B) set for circuits and tunnels using the Adaptive Rate Limiting (ARL) feature. For more information on
ARL and ARL restrictions, refer to Adaptive Rate Limiting on page 20.

Failover circuits and groups

When considering the 10 Gbps bandwidth limit for each DP complex on an FX8-24 blade, you must
also consider failover circuits configured for VE_Ports in each complex. For example, you cannot
create a circuit to use an address for a crossport if a 10 Gbps failover circuit is assigned to the
VE_Port on that crossport. If the failover circuit were to come online, there would be no available
bandwidth for the new circuit.
Failover groups allow you to define a set of metric 0 and metric 1 circuits that are part of a failover
group. When all metric 0 circuits in the group fail, metric 1 circuits take over operation, even if there
are metric 0 circuits still active in other failover groups. Typically, you would configure only one metric 0
circuit in a failover group. For detailed information, refer to Circuit failover on page 65.
In calculating total bandwidth usage for a tunnel, you must also add the total bandwidth usage per
failover group.
To calculate the total bandwidth usage for the failover group in the tunnel, for each failover group (0
through 9), perform the following steps:
1. Add the consumed bandwidth for all metric 0 circuits in the failover group.
2. Add the total consumed bandwidth for all metric 1 circuits in the failover group.
The greater of the two values is the total bandwidth usage for the failover group in the tunnel.

Bandwidth allocation example

The basis for all bandwidth calculations is determining how much bandwidth a given tunnel is
consuming. The next step is determining where the tunnel is consuming that bandwidth. You must
consider the 10 Gbps limits for back-end ports, front-end ports, and crossports. A tunnel is at least
using back-end and front-end port bandwidth. If crossport circuits are configured, then it is using
crossport bandwidth as well.
For example, suppose that two 10 Gbps circuits are configured for a tunnel on VE_Port 12. Circuit 0
has a metric of 0 on xge1 and circuit 1 is a failover circuit with a metric of 1 on xge0 (refer to the
illustration under Front-end and back-end bandwidth on page 44). Note that configuring circuit 1 on
xge0 is a crossport configuration. Although this configuration is allowed, you cannot create additional
circuits for VE_Port group 12 through 21 or group 22 through 31 for the following reasons:

46 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade 7840 Extension Switch

• For VE_Port group 12 through 21, VE_Port 12 is consuming the maximum 10 Gbps of allocated
back-end port bandwidth. Refer to Calculating crossport bandwidth on page 46.
• You cannot create a crossport so that VE_Ports 22 through 31 use xge1 because VE_Port 12 is
consuming the maximum 10 Gbps of crossport bandwidth for its failover circuit. Refer to Calculating
crossport bandwidth on page 46.
• If VE_Port 12 fails, all 10 Gbps traffic will flow over the crossport and the xge0 front-end port. If
additional circuits were already configured for the VE_Port 22 through 31 group, the front-end port
bandwidth would exceed the 10 Gbps limit for xge0. Refer to Calculating front-end bandwidth on
page 45.

Brocade 7840 Extension Switch

This section provides information on ports, circuits, and tunnels specific to the Brocade 7840 Extension
Switch.

NOTE
You cannot connect extension tunnels created on a Brocade 7840 switch to interfaces on a Brocade
7500 switch, 7800 switch or FX8-24 blade.
The following figure illustrates the FC ports, 10/1 GbE, and 40 GbE ports on the Brocade 7840 switch.

FIGURE 8 Brocade 7840 switch ports and status indicators

1. System (SYS) status LED

2. Power (PWR) LED
3. USB port
4. Ethernet management (mgmt) port
5. Serial Console management port
6. FC ports 0-23
7. 40 GbE ports 0-1
8. 1/10 GbE ports 2-17
The Brocade 7840 Extension Switch provides 24 16 Gbps FC ports (FC0-FC23) numbered 0-23 on the
switch, two 40 GbE ports (ge0-ge1) numbered 0-1 on the switch, and 16 1/10 GbE ports (ge2-ge17)
numbered 2-17 on the switch. Up to 20 VE_Ports are supported for tunnel configurations. Typically, only
one VE_Port is needed per remote site.

NOTE
The 40 GbE ports are enabled for configuring IP addresses with the 7840 WAN Rate Upgrade 2 license.

Brocade Fabric OS Extension Administrator's Guide 47

53-1003507-05
Brocade 7840 DP components and VE_Port distribution

Brocade 7840 DP components and VE_Port distribution

Each Brocade 7840 supports two data processor (DP) complexes. Each DP complex contains a data
processor (DP) attached to switching ASICs, and consists of special purpose hardware for extension
functions and multicore network processors.
The following figures illustrate components and connections for each DP complex in the Brocade 7840
switch, when the switch is enabled in 10VE or 20VE modes. All 10, 20, and 40 Gbps connections
shown in the illustrations are full-duplex and internal in the switch. For more information about 10VE
and 20VE port modes, refer to 10VE and 20VE port modes on page 52.

NOTE
The following figure applies to the Brocade 7840 switch when it is in FCIP mode and not hybrid mode
for IP Extension.

48 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Extension Features on Brocade Extension Switches and Blades

FIGURE 9 DP components and VE_Port distribution in 10VE Mode

Brocade Fabric OS Extension Administrator's Guide 49

53-1003507-05
Extension Features on Brocade Extension Switches and Blades

FIGURE 10 DP components and VE_Port distribution in 20VE Mode

As shown in the illustrations:

• There is a 40 Gbps full-duplex connection between the FC switching ASIC and external Gen5 FC
ports and each DP.
• Fibre Channel (FC) frames are compressed with the fast deflate compression hardware.
• There are two 10 Gbps full-duplex connections from the fast deflate compression hardware to the
VE_Ports and from the VE_Ports to the DP network processor. These 10 Gbps connections can
accommodate multiple 10 Gbps or less bandwidth tunnels; however, the maximum bandwidth size
of any one tunnel across these internal connections can be no more than 10 Gbps.
• From the network processors, data can be encrypted by the IPsec hardware using high-speed low-
latency hardware-based encryptors. Each DP network processor can produce 20 Gbps of data flow
going towards or coming from the external Ethernet interfaces and the WAN.
If a 4:1 compression ratio is achieved using fast deflate compression, then 80 Gbps is available to
external FC ports. The Adaptive Rate Limiting (ARL) aggregate of all circuit maximum values on a
single DP complex cannot exceed 40 Gbps. The ARL aggregate of all circuit minimum values for a

50 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 10 GbE and 40 GbE port considerations

single DP complex cannot exceed 20 Gbps. All circuits includes all circuits from all tunnels, not just all
circuits from a single tunnel.

NOTE
Typical deflate compression may achieve different compression ratios. Brocade makes no promises as
to the achievable compression ratios for customer-specific data.
The VE_Port that you use for configuring the tunnel also selects the DP complex that will be used for
processing. The following lists VE_Port distribution on each DP complex for 10VE and 20VE modes.
Refer to 10VE and 20VE port modes on page 52 for more information.
• DP0
‐ 10VE: VE_Ports 24-28
‐ 20VE: VE_Ports 24-33
• DP1
‐ 10VE: VE_Ports 34-38
‐ 20VE: VE_Ports 34-43
For additional specifications and requirements for 7840 switch ports, tunnels, and circuits, refer to
Brocade 7840 extension switches on page 64.

10 GbE and 40 GbE port considerations

Enhanced 10 GbE and 40 GbE port operation requires special considerations when configuring circuits,
tunnels, failover operations, and bandwidth.
For a complete list of tunnel, circuit, and IP address requirements and capacities, refer to Tunnel and
circuit requirements on page 61.

Multigigabit circuits
On a single Brocade 7840 VE port, each tunnel you create is limited to a maximum of eight (8) circuits.
The maximum committed rate of a single circuit is 10 Gbps, whether configured on a 10 GbE or 40 GbE
port.

NOTE
There is no difference in latency or throughput performance for single or multigigabit circuits.

Port grouping
The Brocade 7840 supports eight groups of Ethernet ports. Specific recommendations can be applied to
ports within a group to help alleviate traffic congestion problems.
Switch Ethernet ports are numbered from left to right, starting with the 40 GbE ports as 0-1. The 10 GbE
ports are numbered 2-17. Refer to the illustration of the switch's port side in Brocade 7840 Extension
Switch on page 47 for port numbering. Port numbers contained in port groups are shown in the
following table.

Brocade Fabric OS Extension Administrator's Guide 51

53-1003507-05
10VE and 20VE port modes

TABLE 9 Brocade 7840 switch port groups

Port number Port group

0, 1, 13, 17 1

2, 6 2

3, 7 3

4, 8 4

5, 9 5

10, 14 6

11, 15 7

12, 16 8

Note that port group 1 contains the two 40 GbE ports (0 and 1) and 10 GbE ports 13 and 17. The
remaining port groups contain the 10GbE ports from 2-16. Consider the following when using ports
from these port groups:
• A port can block any port in its port group, but it cannot block a port outside of its port group.
• A port could affect another port in the same group due to differences in port speed or if the port is
back-pressured due to Ethernet pause from an external switch.
To avoid these effects on ports within the same port group, it is best that you do not mix speeds for
ports within the group. Recommendations for the port groups are as follows:
• In port group 1, because the 40 GbE ports are fixed at 40 Gbps, either use the 40 GbE ports or the
10 GbE ports at 10 Gbps or 1 Gbps.
• In port groups 2 through 8, which contain all 10 GbE ports, either configure the ports at 10 Gbps or
1 Gbps.

NOTE
The table applies to ports configured in WAN mode. If the ports are configured as LAN ports, the
grouping and blocking does not apply. As a recommended best practice, allocate a LAN port out of the
same group as a WAN port.

10VE and 20VE port modes

You can configure the Brocade 7480 switch in either 10VE mode (default) or 20VE mode using the
extncfg command. This command is disruptive as it requires rebooting the switch.
In 10VE mode, 10 of the 20 VE_Ports are disabled. These are VE_Ports 29-33 and 39-43. Five
VE_Ports are enabled on each DP complex as follows:

52 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Extension Hot Code Load

• DP0 - VE_Ports 24-28

• DP1 - VE_Ports 34-38
In 10VE mode, a VE_Port can use all Fibre Channel bandwidth available to the DP complex where it
resides, a maximum of 20 Gbps.
In 20VE mode, all 20 VE_Ports are enabled, 10 on each DP complex as follows:
• DP0 - VE_Ports 24-33
• DP1 - VE_Ports 34-43
In 20VE mode, a single VE_Port on a DP complex can use half the Fibre Channel bandwidth available
to the DP complex where it resides, a maximum of 10 Gbps. This option allows use of more VE_Ports,
but at a lower maximum bandwidth.
To configure these modes, refer to the Configuring Extension Features chapter.

Extension Hot Code Load

Extension Hot Code Load (Extension HCL) allows nondisruptive firmware updates on the Brocade 7840
extension switch for FC traffic over the extension tunnels. HCL benefits mainframe environments by
supporting nonstop applications such as data replication and tape backups. Extension HCL maintains
device to mainframe connectivity while the firmware downloads, without disrupting active I/O. Note that
when the switch is operating in hybrid mode, HCL is disruptive for IP traffic.
Fabric OS 7.4.0 provides support for concurrent upgrades from this release to newer releases when
available. Downgrades from Fabric OS 7.4.0 to 7.3 versions should follow the prior recommendation
and perform the updates on one switch at a time.
The Brocade 7840 switch has two DP complexes, referred to as DP0 and DP1 (refer to Brocade 7840
DP components and VE_Port distribution on page 48). An Extension HCL firmware update occurs on
one DP complex at a time. When a firmware update is initiated, the process always starts on DP0.
Before DP0 is updated to the new firmware, traffic fails over to DP1 to maintain communication between
the local and remote switch.
Extension HCL uses three tunnels groups contained within the extension tunnel, as shown in the
following figure, to perform the nondisruptive firmware upload process. The local backup tunnels (LBTs)
and remote backup tunnels (RBTs) are automatically created only when you configure an HCL
compliant tunnel. Not all tunnels are HCL capable. You must configure at least 2 new IP addresses that
will be used for the LBT and RBT tunnels, and then assign those addresses to one of the circuits as the
local HA IP address and remote HA IP address:
• The main tunnel (MT) group provides connectivity during normal operations.
• A local backup tunnel (LBT) group maintains connectivity from the remote switch when the local
switch DP0 is being upgraded. This tunnel, dormant during non-Extension HCL operations, is created
automatically on the local DP1.
• A remote backup tunnel (RBT) maintains connectivity from the local switch when the remote switch
DP0 is being upgraded. This tunnel group, dormant during non-Extension HCL operations, is created
automatically on the remote DP1 when the corresponding local HA IP address is defined.

Brocade Fabric OS Extension Administrator's Guide 53

53-1003507-05
Extension Features on Brocade Extension Switches and Blades

FIGURE 11 Extension HCL tunnels

The main tunnel (MT) is what you normally configure to create an extension tunnel from a VE_Port
using the portcfg fciptunnel command and appropriate tunnel and circuit parameters. The MT carries
traffic through the extension tunnel to the remote switch. The LBT is created upon specifying the local
HA IP address for the circuit, and the RBT is created upon specifying the remote HA IP address for the
circuit. All three tunnel groups (MT, LBT, and RBT) are associated with the same VE_Port.
When an extension tunnel is configured to be HCL capable, the LBT and RBT tunnel groups are
always present. These connections are established at switch boot up or when the tunnel and circuits
are created.
These tunnel groups are utilized in the following Extension HCL upgrade process:
1. The firmware writes to the backup partition of the control processor.
2. The control processor reboots from the backup partition with the new firmware.
3. The local DP0 is updated with the new firmware using the following process.
a. Perform feature disable processing on the MT on DP0.
b. Traffic from the MT is rerouted to DP1 through the LBT so that data traffic can continue between
the switches. In-order data delivery is maintained.
c. DP0 reboots with the new firmware and the configuration is reloaded.
d. Traffic from the LBT is failed-back to DP0 through the MT.
4. The local DP1 is updated with new firmware using the following process.
a. Perform feature disable processing on the MT on DP1.
b. Traffic from the MT is rerouted to DP0 through the LBT so that data traffic can continue between
the switches. In-order data delivery is maintained.
c. DP1 reboots with the new firmware and the configuration is reloaded.
d. Traffic from the LBT is failed-back to DP1 through the MT.
5. After firmware is updated on DP1 and all MTs, LBT, and RBT are online, the Extension HCL
firmware update is complete.
During the update process, tunnels and trunks change state (up or down). The MT provides
connectivity during normal operations. It is up during normal operation and down only during the
Extension HCL process. The RBT and LBT are normally up during normal operation, but do not handle
traffic. They operate to handle traffic during the Extension HCL process. RBT handles traffic when the
remote switch DP0 undergoes the Extension HCL process. The RBT is visible as a backup tunnel on
local DP0.

NOTE
High availability (HA) tunnel groups are not supported by IP Extension. This means hot code load
(HCL) is disruptive to IP traffic during HCL but not to FC traffic when the Brocade 7840 is in hybrid
mode.
To configure Extension HCL, refer to Configuring Extension HCL on page 86.

54 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Extension HCL Limitations and considerations

Extension HCL Limitations and considerations

Following are limitations and considerations for using the Extension HCL feature on the Brocade 7840
switch:
• No configuration changes are permitted during the Extension HCL process. This includes modifying
tunnel or circuit parameters. New device connections that require zone checking can possibly
experience timeouts during the CP reboot phase of the firmware download. The CP performs all
zone checking and therefore must be active to process new SID/DID connection requests, such as
PLOGIs.
• Extension HCL supports Virtual Fabrics (VF) and FC Routing (FCR with the IR license) and all
existing features.
• Extension HCL was designed for all environments including mainframe FICON XRC and tape and
open systems disk replication (EMC SRDF, HDS Universal Replicator, IBM Global Mirror, HP Remote
Copy, and others). Extension HCL supports asynchronous and synchronous environments.
• The Brocade 7840 switch has two data processor (DP) complexes: DP0 and DP1. During the HCL
process, each DP reloads one at a time, while the other DP remains operational. Consider the
following for planning and use of the switch during this process:
‐ Because only one DP complex remains operational at a time, the total switch capacity is
temporarily diminished by 50 percent.
‐ FCIP data is not lost and remains in order. Extension HCL does not cause FICON interface control
check (IFCC). When the Brocade 7840 is operating in hybrid mode, IP traffic will be disrupted.
‐ The use of Extension HCL requires proper planning. There is large amount of bandwidth available
as the Fibre Channel (FC) and FICON side of the switch provides 80 Gbps. In addition, there are
typically A and B paths for a total of 160 Gbps in a redundant replication network. This is more
than enough bandwidth for most replication networks, even during a path failure or firmware
update. Apportioning bandwidth to one DP complex or using only 50 percent of the capacity
across both DP complexes reserves adequate bandwidth for high-availability operations. This is
considered best practice.
‐ The aggregate of all FC and FICON application data passing through the Brocade 7840 cannot
exceed 20 Gbps per DP complex multiplied by the achievable compression ratio, or 40 Gbps,
whichever is smaller. For example, if 2:1 compression can be achieved, then the storage
application could maintain 40 Gbps of throughput across the Extension connection. This is true for
both 10VE and 20VE operating modes.
• Although most firmware updates will support Extension HCL, not every Fabric OS release will
guarantee firmware capable of using this feature. Refer to the Fabric OS release notes for details.
• The firmware on the switch at each end the tunnel must be compatible. If not, this will prevent
successful tunnel formation when the main tunnel attempts to come back online or could introduce
instability and aberrant behavior.
• Extension HCL does not require any additional communication paths. Although there will be existing
FC and tunnel connections used for the normal operation of data replication and tape backup, this is
the only requirement.
• Extension HCL is exclusive to the Brocade 7840. It is not compatible with the Brocade 7500 switch,
7800 switch, or the FX8-24 blade.
• Just before the DP complex is reset during the upgrade process, an FTRACE capture is triggered in
the event that this information is needed post-reset.
• Extension HCL takes advantage of RASlog warnings and error messages (WARN/ERROR).
• If parallel tunnels are configured between local and remote sites, you must enable LLL and set the
VE link cost to static. Otherwise, FC traffic might be disrupted during HCL activity.
• When Teradata Emulation is enabled on an Extension tunnel, Extension HCL is not supported. You
must perform a disruptive firmware download.

Brocade Fabric OS Extension Administrator's Guide 55

53-1003507-05
IP Extension

NOTE
When the Brocade 7840 is operating in hybrid mode, HCL is disruptive to the IP traffic.

IP Extension
The Brocade 7840 switch supports IP Extension. IP Extension provides layer 3 extension for IP
storage replication. The Brocade 7840 acts as the gateway for the LAN, but there is no layer 2
extension, which means each side of the network must be on a different subnet.
The extended IP traffic receives the same benefits as does traditional FCIP traffic.
• Compression
• High speed encryption
• Frame based load levelling and lossless failover
• Network bandwidth management through rate shaping and QoS
IP Extension requires that you configure the Brocade 7840 to operate in hybrid mode. Configuring
hybrid mode is disruptive because a reboot is required to load the hybrid mode image. Internal
connections are remapped to provide 20Gbps of LAN traffic and 10Gbps of FC traffic. A maximum of
20Gbps of WAN traffic is supported.
When in hybrid mode, the Brocade 7840 allows up to 8 of the 10GbE ports to be configured as LAN
ports. LAN ports are not grouped, as opposed to WAN ports which are grouped. LAN ports do not
block each other, and LAN ports do not block WAN ports. The recommended best practice is to pick
one port for each port group to be a LAN port.

Tunnels and hybrid mode

A tunnel on a VE_Port must be configured to enable IP Extension. A tunnel that supports IP traffic
provides additional QoS priorities for IP Extension. The tunnel can carry both FC and IP traffic. When a
tunnel is running in FC-only mode, it is compatible with a Brocade 7840 running in FCIP mode (that is,
not in hybrid mode). FC traffic will be limited to 10Gbps.
Compression is supported on a tunnel in hybrid mode. With FC traffic, all compression modes are
supported: fast deflate, aggressive deflate, and deflate compression. IP traffic is limited to two
compression modes: aggressive deflate and deflate compression.

Out of order delivery on a tunnel

Head of line blocking (HoLB) is mitigated for the extended IP flows through the tunnel. Each flow in a
TCP connection receives an independent stream in the tunnel. The data must be delivered in-order for
the stream; however, data can be delivered out-of-order for the tunnel. This allows the WAN to pass up
any data that is received out of order because packet loss recovery is isolated on the WAN to the
impacted stream, or connection, that the lost data belongs to.

IP Extension and IP on the LAN

IP Extension supports up to eight IP addresses per DP complex for LAN traffic. IPv4 and IPv6 traffic
are supported. Each DP complex keeps its own LAN interface as well as its own MAC for LAN traffic.
In this manner, the DP complex acts as a switch virtual interface (SVI) for the LAN.

56 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 IP Extension and traffic control lists

All GbE ports that are configured as LAN ports can access the SVI addresses of each DP complex. This
allows for multiple GbE ports to access a single IP gateway. In addition, link aggregation groups (LAGs)
are supported. A single LAG can contain up to four ports. A total of eight LAGs are supported.

NOTE
In the FOS 7.4.0 release, only static LAGs are supported.

Jumbo frames are supported for LAN traffic. A jumbo frame can be up to 9216 bytes.
VLAN tagging is supported in IP Extension. Stacked tagging (IEE802.1ad) is not supported.

IP Extension and traffic control lists

Traffic control list (TCL) configurations provide LAN traffic mapping to specific tunnels. With a TCL, you
create rules that identify specific characteristics of the LAN traffic, such as IP addresses, layer 4
protocols and ports, VLAN tags, and so on. For each rule you create, you can allow or deny that specific
traffic type. When traffic is allowed, the TCL rule specifies which tunnel and which QoS to use for that
traffic type. If the rule denies traffic, it specifies which DP complex (DP0 or DP1) to apply the rule to.
NOTE: You must create at least one TCL rule. The default rule, which cannot be modified, is to deny all
traffic.
By using TCL, LAN traffic is mapped to a tunnel and QoS.

IP Extension and QoS

Three fabric QoS priorities are supported for FC, which are high, medium, and low. You can configure
the QoS distribution or use the default distribution of 50% for high, 30% for medium, and 20% for low.
For LAN traffic through IP Extension, three additional QoS priorities are created and default distributions
are provided. The three priorities are IP-High, IP-Medium, and IP-Low. The QoS is added when a tunnel
is enabled for IP Extension. Again, you can change the QoS configuration or use the default.
With IP Extension introduced in FOS release 7.4.0, a second level of QoS is introduced, called QoS
groups. The IP group is for IP Extension traffic. The FC group is for fibre channel (or FCIP) traffic. This
allows you to prioritize your traffic between FC and IP as needed. For example, you can specify a QoS
distribution group ratio of 60% FC and 40% IP. The default group distribution is 50% FC and 50% IP. In
other words, QoS distributions are specified separately for FC traffic and IP traffic when the Brocade
7840 is operating in hybrid mode.

NOTE
Minimum allocation for a single QoS type (high, medium, low) should be 10%. QoS allocations within a
group must total 100%. In addition, allocation for either FC or IP cannot exceed 90%.

IP Extension and compression

The Brocade 7840 supports four compression levels:
• None
• Fast deflate (fast deflate is not supported for IP traffic in hybrid mode)
• Deflate
• Aggressive deflate
Compression can be configured at the QoS group / protocol level when the Brocade 7840 is operating
in hybrid mode. Different compressions can be applied to FC extension traffic and IP extension traffic on

Brocade Fabric OS Extension Administrator's Guide 57

53-1003507-05
IP Extension and IP LAN deployment

the same tunnel. With the protocol selection, you can use fast deflate for FC traffic while the IP traffic
is using deflate or aggressive deflate compression.
Compression is configured at the tunnel level, for all traffic on FC and IP protocols, but you can
override the tunnel settings and select different compression at the QoS group / protocol level.

IP Extension and IP LAN deployment

IP Extension is supported only in topologies with directly-connected devices. This allows IP storage to
be directly connected to the Brocade 7840 LAN ports, or the IP storage can be connected by means of
layer 2 switches to the Brocade 7840 LAN ports.
Link aggregation groups (LAGs) are supported between the Brocade 7840 LAN ports and a layer 2
switch.
In the FOS 7.4.0 release, NIC teaming or other similar methods are not supported. Also, policy-based
routing redirection is not supported.
When using IP storage arrays, the IP storage is connected either to a layer 2 switch or directly to a
Brocade 7840. The following figures show LAN deployment as direct connect and as layer 2 switch
connect.

FIGURE 12 IP Extension direct connect to IP storage

As a guideline, configure the IP storage array with the SVI on one of the DP complexes as the next
hop gateway. Based on the next hop configuration, the storage device will learn the MAC address of
the Brocade 7840 SVI IP address through an ARP or Neighbor Discovery Protocol (NDP) request.
When IP storage devices are connected to a layer 2 switch, as shown in the following figure, you can
use LAGs to connect to the Brocade 7840 LAN ports. The maximum number of ports in a LAG group
is four. The maximum number of LAG groups is 8. Make sure there is only one path between a single
layer 2 domain and the Brocade 7840.

NOTE
In the FOS 7.4.0 release, static LAG is supported.

58 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 IP Extension limitations and considerations

FIGURE 13 IP Extension direct connect to L2 switch

As shown in the figure, the Brocade 7840 switch ports are connected to the IP storage array by means
of the layer 2 switch and the WAN ports are connected to the WAN gateway. You must configure at
least one SVI LAN IP address for each DP complex that is used.
The router can be used as the WAN gateway but it cannot be used on the LAN unless it is acting as a
layer 2 device.

IP Extension limitations and considerations

For TCP traffic, the following considerations apply:

• Flow control for the LAN TCP sessions is handled by means of internal management of the receive
window. The receive window allows up to 256 kB of data onto the LAN. However, the window will
degrade to 0 if the WAN becomes congested, or the target device experiences slow draining.
• TCP connections are limited to 512 connections per DP complex. Additional connection requests are
dropped.
• Up to 512 TCP open connection requests per second are allowed. Additional open connection
requests are dropped. By limiting the number of connection requests per second, this helps prevent
Denial of Service (DoS) attacks.
• Statistics are available for the number of dropped connections.

License options
Important Extension features and FICON extension capabilities of the Brocade 7840 switch require the
feature licenses shown in the following table. Use the licenseshow command to display license keys
and licenses currently installed.

Brocade Fabric OS Extension Administrator's Guide 59

53-1003507-05
Path Maximum Transmission Unit discovery

TABLE 10 Brocade 7840 feature licenses

Feature Purpose License (licenseShow

output)

WAN Rate Upgrade 1 Increases bandwidth available to all extension tunnels WAN Rate Upgrade 1
configured on the switch from 5 Gbps for the base hardware license
to 10 Gbps.

WAN Rate Upgrade 2 Allows unlimited bandwidth for all tunnels configured on the WAN Rate Upgrade 2
switch. This also enables the 40 GbE ports so that they can license
be used for configuring IP addresses.

NOTE
You must have a WAN Rate Upgrade 1 license to activate
the WAN Rate Upgrade 2 license.

Advanced FICON Enables accelerated tape read/write and IBM z/OS Global Advanced FICON
acceleration Mirror, Teradata, and printer emulation features in FICON Acceleration (FTR_AFA)
environments. Slot-based license. license

Advanced Extension This is enabled on the Brocade 7840 switch at the factory. Advanced Extension
License Required for multiple-circuit tunnels, Trunking, ARL. (FTR_AE) license

For complete information about the licenses described in the preceding table and additional licenses
available for the Brocade 7840 switch, refer to the Fabric OS Software Licensing Guide.

Path Maximum Transmission Unit discovery

Path Maximum Transmission Unit (PMTU) discovery is supported on Brocade 7840 Extension
Switches. On Brocade switches, PMTU is the process of sending Internet Control Message Protocol
(ICMP) datagrams of various known sizes across an IP network to determine the supported maximum
datagram size.
Based on the largest ICMP Echo Reply datagram received, the PMTU discovery process sets the IP
MTU for that circuit's IP interface (ipif). Each circuit initiates the PMTU discovery process prior to
coming online. This is required because the circuit may have gone offline due to a link failure, rerouted
to a new path, and now has a different MTU. If a circuit bounces, the PMTU discovery process will be
initiated when attempting to re-establish the circuit. The PMTU discovery process can result in more
time for the circuit establishment. The smallest supported MTU size is 1280 bytes. The largest
supported IP MTU size on the Brocade 7840 is 9216 bytes; however, PMTU discovery will not
discover an MTU greater than 9100 bytes. If the IP network's MTU is known, the best practice is to set
it manually in the portcfg ipif command. This will avoid values determined by PMTU discovery that
are less than the exact MTU of the IP network.

60 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Tunnel and circuit requirements

PMTU requires that ICMP is permitted across all IP network devices and the WAN. A rudimentary check
would be if you could ping devices across this network. Brocade PMTU discovery uses ICMP Echo
Requests. In most cases, only a firewall would block ICMP. If there are no firewalls most likely ICMP is
free to traverse the network. If PMTU discovery cannot communicate with the peer switch, the circuit will
not be established.
Enable PMTU discovery by setting the MTU value to "auto" when configuring the ipif for a circuit using
the portcfg ipif command. Use the portshow ipif command to show the configuration of the MTU
parameter and portshow fcipcircuit --detail command to display the actual discovered PMTU value
being used. You can also initiate PMTU discovery using the portcmd --pmtu command.

Tunnel and circuit requirements

This section describes tunnel and circuit characteristics, capacities, restrictions, and usage on Brocade
extension switches and blades.

General tunnel, circuit, and port requirements

• You can define multiple addresses on Ethernet ports to configure multiple circuits. Multiple circuits
can be configured as a trunk, which provides multiple source and destination addresses to route
traffic across an IP network, provide load leveling, and provide failover capabilities.
• The committed rate for a circuit associated with a physical port cannot exceed the rate of the port.
• In a scenario where a tunnel has multiple circuits of different metrics (0 or 1), circuits with higher
metrics (1) are treated as standby circuits and are only used when all lower metric (0) circuits fail.
Using Circuit Failover Grouping, you can better control which metric 1 circuits will be activated if a
metric 0 circuit fails.
• A circuit defines source and destination IP addresses on either end of a tunnel.
• If the circuit source and destination IP addresses are not on the same subnet, an IP static route
(iproute) must be defined on both sides of the tunnels that designates the gateway IP addresses.
• As a best practice, all tunnel and circuit settings should be identical on both sides of the tunnel. This
includes committed bandwidth, IPsec, compression, ARL minimum and maximum, Fastwrite, OSTP,
FICON tunnel, and keepalive timeout values (KATOV). You must configure the tunnel and circuit
parameters correctly on both sides of the network, otherwise the tunnel or circuit will fail to come
online.
• VE_Ports or VEX_Ports cannot connect in parallel to the same domain at the same time as Fibre
Channel E_Ports or EX_Ports.
• When load-leveling across multiple circuits, the difference between the ARL minimum data rate set
on the slowest circuit in the trunk and the fastest circuit should be no greater than a factor of four. For
example, a 100 Mbps circuit and a 400 Mbps circuit will work, but a 10 Mbps and a 400 Mbps circuit
will not work. This ensures that the entire bandwidth of the trunk can be utilized. If you configure
circuits with the committed rates that differ by more than a factor of four, the entire bandwidth of the
trunk cannot be fully utilized.
For more information on tunnel and circuit requirements, refer to ARL configuration limitations on page
21.

Brocade 7800 extension switches

The Brocade 7800 extension switch has the following requirements for tunnels, circuits, and ports.
IP addresses and routes:

Brocade Fabric OS Extension Administrator's Guide 61

53-1003507-05
Extension Features on Brocade Extension Switches and Blades

• You can define up to eight IP addresses for a GbE port.

• You can define up to 32 IP routes for each GbE port.
VE_Ports, VEX_Ports. and EX_Ports:
• The switch can support eight VE_Ports. VE_Ports are numbered from 16 through 23, therefore up
to eight tunnels can be created. Each tunnel is identified with a VE_Port number.
• The switch supports EX_Ports and VEX_Ports to avoid the need to merge fabrics.
• VE_Ports are not associated with a particular GbE port.
Bandwidths, maximum and minimum rates:
• The minimum committed rate for a circuit is 10 Mbps.
• As a best practice, Fibre Channel traffic through all VE_Port tunnels should not exceed limits set by
Adaptive Rate Limiting (ARL). For example, if the link is 500 Mbps, the aggregate of the ARL
maximum rates connected to that WAN link can be no more than 500 Mbps. For ingress rates, there
is no limit because the FC flow control (BBC) rate limits the incoming data.
• For ARL, configure minimum rates of all the tunnels so that the combined rate does not exceed 6
Gbps for all VE_Ports or the aggregate does not exceed the available WAN bandwidth.
• The maximum trunk capacity is 6 Gbps.
Circuits:
• The switch contains up to six GbE ports. You can configure up to six circuits per tunnel (VE_Port)
spread out over any of these ports.
• A limit of four circuits can be configured on a single GbE port. Each circuit requires a unique IP
address pair.
• The total circuits per switch cannot exceed 24 (a total of four circuits for all GbE ports).
• A single circuit cannot exceed 1 Gbps capacity.
Although a Brocade 7800 switch only contains six GbE ports, eight VE_Ports (16-23) are available for
tunnels. This is because in most cases one VE_Port is used per site and there may be multiple sites
connected to the switch. As another case, when using logical fabrics, a VE_Port can be used per
logical switch. The VE_Ports in different logical switches can share a single GbE port located in the
default switch. Refer to the example distribution in the following table for how the Brocade 7800
VE_Ports GbE ports might be used for port sharing in circuit configurations. Note that no more than
four VE_Ports (tunnels) are using the same GbE port, which is the standard limit for this switch.

TABLE 11 Example VE_Ports versus GbE ports used on the Brocade 7800 switch

VE_Port GbE Ports

16 GE0, GE1, GE2, GE3

17 GE0, GE1, GE2, GE3

18 GE0, GE1, GE2, GE3

19 GE0, GE1, GE2, GE3

62 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade FX8-24 extension blades

TABLE 11 Example VE_Ports versus GbE ports used on the Brocade 7800 switch (Continued)

VE_Port GbE Ports

20 GE4, GE5

21 GE4, GE5

22 GE4, GE5

23 GE4, GE5

Refer to Ethernet port sharing on page 96 for more information on port sharing in a Virtual Fabrics
environment.

Brocade FX8-24 extension blades

The Brocade FX8-24 has the following requirements and specifications for tunnels, circuits, and ports.
IP addresses and routes:
• You can define up to 10 IP addresses for a 10 GbE port and an additional 10 addresses on
crossports when operating in 10 Gbps mode.
• You can define up to eight IP addresses for a 1 GbE port.
VE_Ports, VE_Port groups, VEX_Ports:
• A Brocade FX8-24 blade can support 20 VE_Ports, and therefore 20 extension tunnels.
• There are two VE_Port groups. DP1 controls ports numbered 12 through 21 and DP0 controls ports
numbered and 22 through 31.
• Each tunnel is identified with a VE_Port number.
• VE_Ports are not associated with a particular Ethernet port.
• The blade also supports VEX_Ports to avoid the need to merge fabrics.
• VE_Port versus Ethernet port usage depends on the blade operating mode as follows:
‐ 1 Gbps mode: VE_Ports 12 through 21 are available to use GbE ports 0 through 9. VE_Ports
22-31, xge0, and xge1 are not available.
‐ In 10 Gbps mode, VE_Ports 12 through 21 are available to use xge1; VE_Ports 22 through 31 are
available to use xge0. GbE ports 0 through 9 are not available.
‐ In 10 Gbps mode, you can also configure VE_Ports 12 through 21 to use port xge0 as a crossport
and VE_Ports 22 through 31 to use port xge1 as a crossport.
‐ In dual mode, VE_Ports 12 through 21 are available to use GbE ports 0 through 9; VE_Ports 22
through 31 are available to use xge0. Port xge1 is not available.
Circuits:

Brocade Fabric OS Extension Administrator's Guide 63

53-1003507-05
Brocade 7840 extension switches

• A limit of 20 circuits can be configured per VE_Port group (12 through 21 or 22 through 31) when
using a 10 GbE port. For the 20 circuits, 10 are configured on local ports and 10 on crossports
• You can configure up to 10 circuits for a trunk (VE_Port).
• The FX8-24 blade contains two 10 GbE ports. You can define up to 10 circuits per trunk spread
across the 10 GbE ports.
• A limit of 10 circuits can be configured on a single 10 GbE port. Each circuit requires a unique IP
address.
• The blade contains ten 1 GbE ports. You can define up to 10 circuits per trunk spread across the
GbE ports.
• A limit of four circuits can be configured on a single 1 GbE port. Each circuit requires a unique IP
address.
Bandwidths, maximum and minimum rates:
• For an FX8-24 blade with a VE_Port group on a 10 GbE port, the sum of the maximum committed
rates of that group's circuits cannot exceed 10 Gbps.
• For ARL, configure minimum rates of all the tunnels so that the combined rate does not exceed 20
Gbps for all VE_Ports on the blade.
• For ARL, you can configure maximum rate of 10 Gbps for all tunnels over a single 10 GbE port and
10 Gbps for any single circuit.
• The minimum committed rate for a circuit is 10 Mbps.
• A circuit between 1 GbE ports cannot exceed the 1 Gbps capacity of the interfaces rate.
For additional considerations on multigigabit circuits configured on 10 GbE ports, refer to Multigigabit
circuits on page 41.

Brocade 7840 extension switches

This section lists requirements and specifications for tunnels, circuits, and ports on Brocade 7840
extension switches.
IP addresses and routes:
• You can configure maximum 60 IP addresses per DP complex.
• You can define up to 128 routes per GbE port; however, you can only define 120 IP routes per DP.
For example, you can configure 64 IP routes defined on ge2.dp0 and another 64 IP routes defined
on ge2.dp1.
VE_Ports, VE_Port groups, and VEX_Ports:
• You can have a maximum 20 VE_Ports on the switch. In the default 10 VE mode, only 10 VE_Ports
are enabled. In 20 VE mode, all 20 VE_Ports are enabled.

NOTE
When the switch operates in hybrid mode, 20 VE mode is not allowed.
• There are two VE_Port groups in 10VE mode. DP0 controls VE_Ports 24-28 and DP1 controls
VE_Ports 34-38. The remaining VE_Ports 29-33 and 39-43 are disabled. Each port group can share
20 Gbps.
• There are four VE_Port groups in 20VE mode. DP0 controls VE_Ports 24-28 and VE_Ports 29-33.
DP1 controls VE_Ports 34-38 and VE_Ports 39-43. Each port group can share 10 Gbps.
• VE_Ports are not associated with a particular Ethernet port.
• VE_Ports cannot connect in parallel to the same domain at the same time as Fibre Channel
E_Ports or EX_Ports.
• VEX_Ports are not supported on this platform.
Bandwidths, maximum and minimum rates:

64 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Circuit failover

• For a VE_Port group, the sum of the minimum committed rates of that group's circuits cannot exceed
10 Gbps when the switch is in 20VE mode and 20 Gbps when the switch is in 10VE mode.
• The minimum committed rate for all VE_Ports in one DP complex cannot exceed 20 Gbps. The
maximum rate for all VE_Ports in one DP complex cannot exceed 40 Gbps.
• The minimum committed rate for a circuit is 20 Mbps.
• The maximum committed rate for a circuit is 10 Gbps.
• With compression, total bandwidth cannot exceed 80 Gbps (40 Gbps per DP) on the Fibre Channel
side.
• The difference between the guaranteed (minimum) and maximum bandwidth for a tunnel cannot
exceed the 5:1 ratio.
Circuits:
• There is no limit for the number of circuits that you can configure on an Ethernet port. Each circuit
requires a unique IP address pair.
• You can configure a maximum of eight circuits for a trunk (VE_Port).
For additional considerations on multigigabit circuits configured on Ethernet ports, refer to Multigigabit
circuits on page 51.

Circuit failover
Each circuit is assigned a metric, either 0 or 1, which is used in managing failover from one circuit to
another. Trunking with metrics uses lossless link loss (LLL), and no in-flight data is lost during the
failover. If a circuit fails, Trunking first tries to retransmit any pending send traffic over another lowest
metric circuit. In the following figure, circuit 1 and circuit 2 are both lowest metric circuits. Circuit 1 has
failed, and transmission fails over to circuit 2, which has the same metric. Traffic that was pending at the
time of failure is retransmitted over circuit 2. In-order delivery is ensured by the receiving extension
switch or blade.

FIGURE 14 Link loss and retransmission over peer lowest metric circuit

NOTE
Modifying a circuit metric disrupts traffic.
In the following figure, circuit 1 is assigned a metric of 0, and circuit 2 is assigned a metric of 1. Both
circuits are in the same tunnel. In this case, circuit 2 is not used until no lowest metric circuits are
available. If all lowest metric circuits fail, then the pending send traffic is retransmitted over any
available circuits with the higher metric. Failover between like metric circuits or between different metric
circuits is lossless.

Brocade Fabric OS Extension Administrator's Guide 65

53-1003507-05
Circuit Failover Grouping

FIGURE 15 Failover to a higher metric standby circuit

Only when all metric 0 circuits fail do available metric 1 circuits cover data transfer. If the metric 1
circuits are not identical in configuration to the metric 0 circuits, then the metric 1 circuits will exhibit a
different behavior. Additionally, if the metric 1 WAN path has different characteristics, these
characteristics define the behavior across the metric 1 circuits. Consider configuring circuit failover
groups to avoid this problem.

Circuit Failover Grouping

With circuit failover groups, you can better control which metric 1 circuits will be activated if a metric 0
circuit fails. To create circuit failover groups, you define a set of metric 0 and metric 1 circuits that are
part of the same failover group. When all metric 0 circuits in the group fail, metric 1 circuits will take
over data transfer, even if there are metric 0 circuits still active in other failover groups.
Typically, you would only define one metric 0 circuit in the group so that a specific metric 1 circuit will
take over data transfer when the metric 0 circuit fails. This configuration prevents the problem of the
tunnel operating in a degraded mode, with fewer than the defined circuits, before multiple metric 0
circuits fail.

Considerations and limitations

Circuit failover groups operate under the following conditions:
• Each failover group is independent and operates autonomously.
• All metric 0 circuits in a group must fail before the metric 1 circuits in that failover group are used.
• All metric 1 circuits in a group are used if all metric 0 circuits in the group fail or there is no metric 0
circuit in the group.
• Circuits can be part of only one failover group
• Circuit failover groups are only supported by Fabric OS v7.2.0 or later.
• Both ends of the tunnel must have the same circuit failover groups defined.
• Tunnel and circuit states will indicate a misconfiguration error if circuit failover group configurations
are not valid.
• Modifying of the failover group ID is a disruptive operation, similar to modifying the metric.
• Circuit failover groups are not used to define load balancing over metric 0 circuits (only failover
rules).
• When no circuit failover groups are defined, failover reverts to the default operation: all metric 0
circuits must fail before failing over to metric 1 circuits.
• A valid failover group requires at least one metric 0 circuit and at least one metric 1 circuit;
otherwise, a warning is displayed.
• The number of valid failover groups defined per tunnel is limited by the number of circuits that you
can create for the switch model as follows:

66 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Examples of circuit failover in groups

‐ For an FX8-24 blade, you can configure up to 5 valid groups on a 10-circuit tunnel.
‐ For a Brocade 7800 switch, you can have up to 3 valid groups because you can configure 6
circuits per tunnel.
‐ For a Brocade 7840 switch, you can have up to 4 valid groups because you can configure 8
circuits per tunnel.
• Consider available WAN bandwidth requirements when configuring failover circuit groups. Refer to
Bandwidth calculation during failover on page 72.

Examples of circuit failover in groups

The following table illustrates circuit failover in a tunnel with two failover groups, each with two circuits.
All data through the tunnel is initially load balanced over circuits 1 and 2. The following occurs during
circuit failover:
• If circuit 1 fails, circuit 3 becomes active and data is load balanced over circuits 2 and 3.
• If circuit 2 fails, circuit 4 becomes active and data is load balanced over circuits 1 and 4.
• If both circuit 1 and 2 fail, circuit 3 and 4 become active and data is load balanced over both circuits.

TABLE 12 Tunnel with two failover groups with two circuits

Circuits in tunnel Failover group ID Circuit bandwidth FSPF link cost if In use for tunnel
circuit goes offline data

Circuit 1 Metric 0 1 500 Mb 1,500 If active, yes.

Circuit 2 Metric 0 2 1000 Mb 1,000 If active, yes.

Circuit 3 Metric 1 1 500 Mb 1,500 Only when circuit 1

fails.

Circuit 4 Metric 1 2 1000 Mb 1,000 Only when circuit 2

fails.

The following table illustrates circuit failover in a tunnel with circuits in failover groups and circuits that
are not part of failover groups. In this configuration, all data is initially load balanced over circuit 1, circuit
2, and circuit 3 (when they are all active). The following occurs during circuit failover:
• If circuit 1 fails, circuit 4 becomes active and data is load balanced over circuit 2, circuit 3, and circuit
4.
Reason: Circuit 1 fails over to circuit 4 (both are in failover group 1) and circuit 3 is active with 500
Mb bandwidth.
• If circuit 2 fails, data is load balanced over circuit 1 and circuit 3, and no other circuit becomes active.

Brocade Fabric OS Extension Administrator's Guide 67

53-1003507-05
Configuring circuit failover groups

Reason: Circuits 1 and 3 are the only active circuits because circuits 4 and 5 only become active
when circuits 1 or 3 fail.
• If circuit 2 and circuit 3 fail, circuit 5 becomes active and data is load balanced over circuit 1 and
circuit 5.
Reason: Ungrouped circuits 2 and 3 fail over to ungrouped circuit 5, which has a metric of 0.
• If circuit 1, circuit 2, and circuit 3 fail, circuit 4 and circuit 5 become active and data is load balanced
over both.
Reason: Circuit 1 fails over to circuit 4, which is the failover circuit for group 1 with a metric of 0.
Ungrouped circuit 5 is the failover circuit for ungrouped, failed circuits 2 and 3.

TABLE 13 Tunnel with failover groups and non-grouped circuits

Circuits in tunnel Failover group ID Circuit bandwidth FSPF link cost if In use for tunnel
circuit goes offline data

Circuit 1 Metric 0 1 500 Mb 1,500 If active, yes.

Circuit 2 Metric 0 Not defined. 500 Mb 1,500 If active, yes.

Circuit 3 Metric 0 Not defined. 500 Mb 1,500 If active, yes.

Circuit 4 Metric 1 1 500 Mb 1,500 Only when circuit 1

fails.

Circuit 5 Metric 1 Not defined. 1000 Mb 1,000 Only when circuits 2

and 3 fails.

Configuring circuit failover groups

Configure failover groups by specifying a metric with the -x|--metric# option and the failover group with
the -g|--failover-group ID option, as in the following commands:
• portcfg fciptunnel slot/ve_port create --remote-ip destination address --local-ip source address -
x|--metric [0/1] -g|--failover-group [0-9]
• portcfg fcipcircuit slot/ve_port create cir# --remote-ip destination address --local-ip source
address -x|--metric [0/1] -g|--failover-group [0-9]
• portcfg fcipcircuit slot/ve_port modify cir# --remote-ip destination address --local-ip source
address -x|--metric [0/1] -g|--failover-group [0-9]

68 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuration examples

Configuration examples
The following example shows the configuration of two failover groups for VE_Port 22 containing two
circuits each. Note that circuit 0 is typically created automatically when the tunnel is created.

portcfg fcipcircuit 8/22 create 0 --remote-ip 1.42.128.93 --local-ip 1.42.128.23 -x 0 -g 0 -b 5000000

-B 5000000
portcfg fcipcircuit 8/22 create 1 --remote-ip 1.42.128.94 --local-ip 1.42.128.24 -x 0 -g 1 -b 2750000
-B 2750000
portcfg fcipcircuit 8/22 create 2 --remote-ip 1.42.128.95 --local-ip 1.42.128.25 -x 1 -g 0 -b 4000000
-B 4000000
portcfg fcipcircuit 8/22 create 3 --remote-ip 1.42.128.96 --local-ip 1.42.128.26 -x 1 -g 1 -b 5000000
-B 5000000

Entering the portshow fciptunnel -c command for the configuration, displays the following output.

-------------------------------------------------------------------------------
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
-------------------------------------------------------------------------------
8/22 - Up cft---- 26m51s 0.00 0.00 1 - -/-
8/22 0 8/xge0 Up ---4--s 26m51s 0.00 0.00 1 5000/5000 0/-
8/22 1 8/xge0 Up ---4--s 26m51s 0.00 0.00 1 2750/2750 0/1
8/22 2 8/xge0 Up ---4--s 2m7s 0.00 0.00 1 4000/4000 1/-
8/22 3 8/xge0 Up ---4--s 0s 0.00 0.00 1 5000/5000 1/1
-------------------------------------------------------------------------------

Note in the output that "-" displays for the group (G) to indicate the default failover group 0 or that no
failover group is configured.
If you do not configure at least one metric 0 and one metric 1 circuit for a failover group, the Opstatus
column of the output will display a failover group warning (FGrpWrn) as in the following output. The
warning occurred because only a circuit with metric 0 was created for failover group 0 and group 1.
Note that FGrpWrn will not be issued for the default group unless a failover group is configured but not
complete with a metric 0 and metric 1 circuit.

NOTE
For the 7840 switch, 'UpWrn' displays if a tunnel is not specified in the portshow fciptunnel command,
such as in portshow fciptunnel all -c -h, while 'Online Warning' displays if a tunnel is
specified, such as in portshow fciptunnel 24.

-------------------------------------------------------------------------------
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
-------------------------------------------------------------------------------
8/22 - FGrpWrn cft---- 20m26s 0.00 0.00 1 - -/-
8/22 0 8/xge0 Up ---4--s 20m26s 0.00 0.00 1 5000/5000 0/-
8/22 1 8/xge0 Up ---4--s 3s 0.00 0.00 2 2750/2750 0/1
-------------------------------------------------------------------------------E

Entering the portshow fciptunnel command for a specific tunnel, such as portshow fciptunnel 8/22 -
c for tunnel 22, displays detailed information for each circuit in the tunnel. Entering the portshow
fcipcircuit command, such as portshow fcipcircuit 8/22 1 for circuit 1, displays detailed information
for the specific circuit. This information includes the failover group ID configured for the circuits. If (Not
Config/Active) displays for Failover Group ID, the default group ID of 0 was used.
For more information on the portcfg fcipcircuit, portcfg fciptunnel, and portshow commands, refer
to the Fabric OS Command Reference.

Brocade Fabric OS Extension Administrator's Guide 69

53-1003507-05
10 GbE Lossless Link Loss (FX8-24 blade)

10 GbE Lossless Link Loss (FX8-24 blade)

Circuit failover is supported between 10 GbE circuits on FX8-24 blades when both 10 GbE ports are
on the same logical switch and are operating in 10 Gbps mode. You can configure higher metric
circuits for failover from lower metric circuits (refer to Circuit failover on page 65). You can also
configure IP addresses for a failover crossport. Crossports are IP addresses (and routes) that belong
to the other 10 GbE port’s VE group. The crossport for xge0 is xge1 and the crossport for xge1 is
xge0. For more information on crossports and configuring crossports, refer to Crossports on page 41.
LLL is supported per VE_Port on the VE_Port's DP complex. Because a VE_Port cannot span GbE
and 10 GbE interfaces, neither can LLL. LLL is supported on both GbE and 10 GbE interfaces, just not
together.
Benefits and limitations of 10 GbE lossless link loss (LLL) failover include the following:
• LLL provides failover to protect against link or network failure and 10 GbE port disable.
• Data will not be lost due to failover.
• Failover supports active-passive and active-active configurations.
• Dual mode is not supported for 10 GbE port failover.
• Failover does not protect against failure of a DP complex.
• Disabling a VE_Port will not use LLL. In this case, route failover will occur at the FC level based on
APT policy, if there is another route available, and may cause loss of FC frames.

NOTE
All circuits and data must belong to a single VE_Port to benefit from LLL.

Configuring failover
There are two types of configuration supported:
• Active-active - Data will be sent on both 10 GbE ports to initiate weighted balancing of the batches
across the trunk circuits.
• Active-passive - Data fails over using LLL to a passive circuit (one with a higher metric) if all active
lower metric circuit paths fail.
You must establish a metric for failover circuits. If no metric is provided, circuit data will be sent
through both ports and the load will be balanced. Circuits have a default metric of 0. A metric of 1 is
required for a standby (passive) circuit.

Active-active configuration

The following example shows an active-active configuration in which two circuits are configured with
the same metric, one circuit going over xge0 and the other circuit going over the crossport using xge1
as the external port. The metric values of both the circuits are the same (default value), so both circuits
send data. The load is balanced across these circuits. The effective bandwidth of the tunnel in this
example is 2 Gbps.
1. Configure an IP address on interface xge0.
portcfg ipif 8/xge0 create 192.168.11.20 netmask 255.255.255.0 mtu 1500
2. Configure an IP address on crossport interface xge1.
portcfg ipif 8/xge1 create 192.168.10.10 netmask 255.255.255.0 mtu 1500 -x
3. Create a tunnel with one circuit going over xge0.
portcfg fciptunnel 8/22 create --remote-ip 192.168.11.20 --local-ip 192.168.11.21
-b 2750000 -B 2750000

70 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Active-passive configuration

4. Add another circuit, going over crossport xge1, to the tunnel.

portcfg fcipcircuit 8/22 create 1 --remote-ip 192.168.10.10 --local-ip
192.168.10.11 -b 1000000 -B 1000000
5. Display local and crossport interface details for xge0.
portshow ipif 8/xge0

NOTE
If the source and destination addresses are on different subnets, you must configure IP routes to the
destination addresses. Refer to Configuring an IP route on page 80.

Active-passive configuration

The following example shows an active-passive configuration in which two circuits are configured with
different metrics, one circuit going over xge0 and the other circuit going over the crossport using xge1
as the external port. In this example, circuit 1 is a failover circuit because it has a higher metric. When
circuit 0 goes down, the traffic is failed over to circuit 1. The effective bandwidth of the tunnel in this
example is 1 Gbps.
1. Configure an IP address on interface xge0.
portcfg ipif 8/xge0 create 192.168.11.20 netmask 255.255.255.0 mtu 1500
2. Configure an IP address on crossport interface xge1.
portcfg ipif 8/xge1 create 192.168.10.10 netmask 255.255.255.0 mtu 1500 -x
3. Create a tunnel with one circuit going over xge0.
portcfg fciptunnel 8/22 create --remote-ip 192.168.11.21 --local-ip 192.168.11.20 -
b 2750000 -B 2750000 --metric 0
4. Add another circuit, going over crossport xge1, to the tunnel.
portcfg fcipcircuit 8/22 create 1 --remote-ip 192.168.10.10 --local-ip
192.168.10.11 -b 1000000 -B 1000000 --metric 1
5. Display local and crossport interface details for xge0.
portshow ipif 8/xge0

NOTE
If the source and destination addresses are on different subnets, you must configure IP routes to the
destination addresses. Refer to Configuring an IP route on page 80.

Failover in TI zones
In Traffic Isolation (TI) zone configurations with failover enabled, non-TI zone traffic will use the
dedicated path if no other E_Port or VE_Port paths exist through the fabric or if the non-dedicated paths
are not the shortest paths. Note that a higher-bandwith tunnel with multiple circuits will become the
shortest path compared to a tunnel with one circuit. A TI zone cannot subvert the Fabric Shortest Path
First (FSPF) protocol. Data will never take a higher cost path because a TI zone has been configured to
do so. It may be necessary to configure explicit link cost to produce Equal-Cost Multi-Path (ECMP) or to
prevent trunk costs from changing in the event that a circuit goes offline.

Brocade Fabric OS Extension Administrator's Guide 71

53-1003507-05
Bandwidth calculation during failover

Bandwidth calculation during failover

The bandwidth of higher metric circuits is not calculated as available bandwidth on a tunnel until all
lowest metric circuits have failed.
Assume the following configurations for circuits 0 through 3:
• Circuits 0 and 1 are created with a metric of 0. Circuit 0 is created with a maximum transmission
rate of 1 Gbps, and circuit 1 is created with a maximum transmission rate of 500 Mbps. Together,
circuits 0 and 1 provide an available bandwidth of 1.5 Gbps.
• Circuits 2 and 3 are created with a metric of 1. Both are created with a maximum transmission rate
of 1 Gbps, for a total of 2 Gbps. This bandwidth is held in reserve.
The following actions occur during circuit failures:
• If either circuit 0 or circuit 1 fails, traffic flows over the remaining circuit while the failed circuit is
being recovered. The available bandwidth is still considered to be 1.5 Gbps.
• If both circuit 0 and circuit 1 fail, there is a failover to circuits 2 and 3, and the available bandwidth is
updated as 2 Gbps.
• If a low metric circuit becomes available again, the high metric circuits return to standby status, and
the available bandwidth is updated again as each circuit comes online. For example, if circuit 0 is
recovered, the available bandwidth is updated as 1 Gbps. If circuit 1 is also recovered, the available
bandwidth is updated as 1.5 Gbps.

72 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
Configuring Extension Features

● Configuration preparation................................................................................................73
● Configuration steps......................................................................................................... 74
● Setting VE_Ports to persistently disabled state.............................................................. 75
● Configuring VEX_Ports................................................................................................... 76
● Configuring the media type for GbE ports 0 and 1 (Brocade 7800 switch)..................... 76
● Setting the GbE port operating mode on FX8-24 blade only.......................................... 76
● Configuring switch and port modes (7840 switch).......................................................... 77
● Configuring port speed (Brocade 7840 switch)............................................................... 79
● Configuring an IPIF......................................................................................................... 79
● Configuring an IP route................................................................................................... 80
● Validating IP connectivity.................................................................................................82
● Creating an Extension tunnel.......................................................................................... 82
● Creating additional circuits ............................................................................................. 85
● Verifying the tunnel configuration.................................................................................... 86
● Configuring Extension HCL.............................................................................................86
● Enabling persistently disabled ports................................................................................90
● Modifying a tunnel........................................................................................................... 90
● Modifying a circuit........................................................................................................... 91
● Deleting an IP interface ..................................................................................................91
● Deleting an IP route ....................................................................................................... 91
● Deleting a trunk............................................................................................................... 92
● Deleting a circuit .............................................................................................................92
● Configuring Per-Priority TCP QoS priorities over a trunk................................................92
● Modifying default priority values......................................................................................93
● Using logical switches..................................................................................................... 94
● Managing QoS, DSCP, and VLANs...............................................................................103
● Implementing IPsec over tunnels.................................................................................. 107
● Traffic Isolation Zoning.................................................................................................. 110

Configuration preparation
Before you begin, do the following:
• Determine the amount of bandwidth that will be required for the remote data replication (RDR),
FICON, or tape application to be deployed.
• Confirm that the WAN link has been provisioned and tested for integrity.
• Make sure that cabling within the data center has been completed.
• Make sure that switches and other devices have been physically installed and powered on.
• Make sure you have admin access to all switches and blades you need to configure.
• For the Brocade 7800 switch, determine if copper or optical ports will be used for GbE ports 0 and 1.
• For the FX8-24 blade, determine which of the three possible GbE or XGE port operating modes will
be used.

Brocade Fabric OS Extension Administrator's Guide 73

53-1003507-05
Configuration steps

• For the FX8-24 blade, determine which 10 GbE crossports to use for active-active or active-passive
configurations.
• For the Brocade 7840 switch, determine the VE_Port operating modes that will be used (10VE
mode or 20VE mode). For information about configuring IP Extension features, refer to
Configuration preparation for IP Extension features on page 111.
• Determine which Ethernet ports will be used. The Ethernet ports on the Brocade 7840 switch are in
groups and connections should be spread across the groups and not within the same group if
possible.
• Obtain subnets and assign IP addresses for each circuit endpoint that you intend to use, plus the
netmask and IP MTU size. The IP MTU size may be smaller than 1500 if there is an IPsec device or
similar device in the path. If the IP MTU is larger than 1500, use the following guidelines for your
extension product:
‐ For the Brocade 7800 switch and FX8-24 blade, use 1500.
‐ For the Brocade 7840 switch, the IP MTU size must be at least 1280. If the supported maximum
IP MTU size in the network is larger than 9216, the IP MTU must be 9216. You can use Path
MTU Discovery to automatically set the IP MTU size for the circuit's IP interface. Refer to
Modifying a tunnel on page 90 for more information.
• Determine the gateway IP address as needed for each route across the WAN. The gateway IP
address will be on the same IP subnet as the subnet used for the IPIF interface that will use that
gateway. The route will be the subnet and netmask on the remote side.
• Determine if there is any reason to turn off selective acknowledgement (SACK). Because SACK
improves performance for most installations, in the Brocade 7800 and FX8-24 it is turned on by
default.
• Determine the VE_Port numbers you want to use. The VE_Port numbers serve as tunnel IDs.
Typically, the first one is used.
• Determine source and destination IP addresses for circuit 0, and the minimum and maximum rates
for ARL. These values are set by the portCfg fciptunnel create command. If ARL is not being
used, then set the minimum and maximum committed rates to the same value.
• Determine how many additional circuits you want to create. You will need the source and
destination IP addresses for each circuit, and the minimum and maximum rates for ARL. You will
need to know if you intend to assign metrics to circuits so that lower metric circuits fail over to
circuits with higher metrics. For all circuits except circuit 0, these values are set by the portCfg
fcipcircuit create command.
• When configuring tunnels to support large numbers of devices, consider memory limitations of the
extension switch or blade if you are enabling any type of emulation feature, such as FCP or FICON.
If too many devices are present or activated at one time, acceleration operations can be negatively
impacted. Refer to Memory use limitations for large-device tunnel configurations on page 27.

Configuration steps
Use the following major steps for configuring extension switches and blades:
1. Persistently disable VE_Ports.
2. If required, configure VEX_Ports.
3. Set the switch operating mode.
• For the FX8-24 blade, set the Gbe and XGbE port operating mode.
• For the Brocade 7840 switch, configure FCIP or hybrid mode.
• If running FCIP mode configure 10VE or 20VE operating mode.
4. Set the GbE port configurations.

74 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Setting VE_Ports to persistently disabled state

• For the Brocade 7800 switch, set the media type for GbE ports 0 and 1.
• For the Brocade 7840 switch, set the GbE port speed to 1G or 10G as needed.
• For the Brocade 7840 switch when running in Hybrid (FCIP + IP-Extension), set the GbE port LAN
port configurations and LAG configurations.
5. Create an IP interface (IPIF) for each circuit that you want on a port by assigning an IP address,
netmask, and an IP MTU size to an Ethernet port using the portCfg ipif command. Refer to
Configuring an IPIF on page 79.
6. Create one or more IP routes to a port if required using the portCfg iproute command. Refer to
Configuring an IP route on page 80.
7. Test the IP connection using the portCmd --ping command.

NOTE
For the Brocade 7800 switch and FX8-24 blade, when using VLANS, VLAN tagging ensures that test
traffic traverses the same path as real traffic. A VLAN tag entry for both the local and remote sides of
the route must exist prior to using the portCmd --ping command.
The 7840 manages the VLANs differently. It automatically supports VLAN tagging if it is specified in
the ipif.
Refer to Managing the VLAN tag table on page 105 for details.
8. Create tunnels using the portCfg fciptunnel command. Refer to Creating an Extension tunnel on
page 82.
9. Create circuits (after circuit 0) and enable or disable features using the portCfg fcipcircuit
command. Refer to Creating additional circuits on page 85.

NOTE
Configuring a tunnel automatically configures circuit 0 for the tunnel, although you can use portcfg
fciptunnel parameters, without any additional parameters, to create a blank tunnel.
10.Persistently enable the VE_Ports.

Setting VE_Ports to persistently disabled state

It is strongly recommended to persistently disable VE_Ports while tunnel configuration is in progress.
This will prevent unwanted fabric merges from occurring until the tunnel is fully configured. You must
change the state of the VE_Ports from persistently enabled to persistently disabled. Once the tunnels
have been fully configured on both ends of the tunnel, you can persistently enable the ports.
1. Enter the portCfgShow command to view ports that are persistently disabled.
2. Enter the portCfgPersistentDisable command to disable any VE_Ports that you will use in the
tunnel configuration.

Disabling ports when FMS Mode is enabled

If you enter portCfgPersistentDisable and receive "command not allowed in fmsmode" or "command
not found" messages, FMS mode may be enabled. You cannot use the portCfgPersistentDisable or
portCfgPersistentEnable commands with FMS mode enabled. Use the portDisable and portEnable
commands instead.
You can determine if FMS mode is enabled by using the ficoncupshow fmsmode command.

Brocade Fabric OS Extension Administrator's Guide 75

53-1003507-05
Configuring VEX_Ports

Configuring VEX_Ports
If you are going to use a VEX_Port in your tunnel configuration, use the portCfgVEXPort command to
configure the port as a VEX_Port. VEX_Ports can be used to avoid merging fabrics over distances in
Extension implementations.
If the fabric is already connected, disable the Ethernet ports and do not enable them until after you
have configured the VEX_Port. This prevents unintentional merging of the two fabrics.
VEX_Ports are described in detail in the Fabric OS Administrator’s Guide. Refer to that publication if
you intend to implement a VEX_Port.
The following example configures a VEX_Port, enables admin, and specifies fabric ID 2 and preferred
domain ID 220.

switch:admin> portcfgvexport 18 -a 1 -f 2 -d 220

Configuring the media type for GbE ports 0 and 1 (Brocade 7800
switch)
Two media types are supported for GbE ports 0 and 1 on the Brocade 7800 switch: copper (RJ-45)
and optical. The media type must be set for GbE ports 0 and 1 using the portCfgGEMediatype
command. The following example configures port 1 (ge1) as an optical ports.

switch:admin> portcfggemediatype ge1 optical

The ge0 option is used for port 0 and the ge1 option is used for port 1. The copper and optical options
are used for the media type.
When you enter this command without specifying the media type, the current media type for the
specified GbE port is displayed, as in the following example.

switch:admin> portcfggemediatype ge1

Port ge1 is configured in optical mode

NOTE
The Optical option references the SFP bays in which optical SFPs are most often used. It is possible
to insert copper-based RJ-45 SFPs into these bays.

Setting the GbE port operating mode on FX8-24 blade only

The GbE ports on an FX8-24 blade can operate in one of three ways:

76 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring switch and port modes (7840 switch)

• 1 Gbps mode: GbE ports 0 through 9 may be enabled as GbE ports, with the XGE ports disabled.
The 10 GbE (FTR_10G) license is not required.
• 10 Gbps mode: 10 GbE ports xge0 and xge1 may be enabled, with GbE ports 0 through 9 disabled.
The 10 GbE (FTR_10G) license is required and must be assigned to the slot in which the FX8-24
blade resides.
• Dual mode: GbE ports 0 through 9 and 10 GbE port xge0 may be enabled, with xge1 disabled. The
10 GbE (FTR_10G) license is required and must be assigned to the slot in which the FX8-24 blade
resides.

NOTE
Switching between 10 Gbps mode and 1 Gbps mode disrupts traffic.

NOTE
Before changing operating modes for a port, you must delete the port’s configuration.
You must configure the desired GbE port mode of operation for the FX8-24 blade using the
bladeCfgGeMode --set mode -slot slot number command. The command options are as follows.

--set mode 1g enables the GbE ports 0 through 9 (xge0 and xge1
are disabled).

10g enables ports xge0 and xge1 (ge0-ge9 ports are

disabled).

dual enables the GbE ports 0 through 9 and xge0 (xge1

is disabled).

-slot slot number Specifies the slot number for the FX8-24 blade.

The following example enables GbE ports 0 through 9 on an FX8-24 blade in slot 8. Ports xge0 and
xge1 are disabled.

switch:admin> bladecfggemode --set 1g -slot 8

You can use the bladecfggemode --show command to display the GbE port mode for the FX8-24
blade in slot 8, as shown in the following example.

bladecfggemode --show -slot 1

bladeCfgGeMode: Blade in slot 1 is configured in 10GigE Mode
10GigE mode: only xge0 and xge1 are enabled (ge0-9 ports are disabled)
The blade in slot 1 supports IP Sec tunnels on only VEs 12 through 21 (xge1 or ge0-
ge9)

Configuring switch and port modes (7840 switch)

You can configure the Brocade 7840 in either FCIP mode (default) or hybrid mode (FCIP and IP
Extension) using the extncfg --app-mode command. This command is disruptive and requires
rebooting the switch.
You can configure the following switch modes:

Brocade Fabric OS Extension Administrator's Guide 77

53-1003507-05
Configuring Extension Features

• FCIP mode—In this mode, only FCIP traffic is sent over the extension tunnels. It allows up to 20G
of FC traffic (80G compressed with 4:1 compression). FCIP mode allows you to chose between
10VE mode and 20VE port modes.
• Hybrid mode—In this mode, FCIP traffic and IP traffic can be sent over the extension tunnels.
Hybrid mode allows for up to 10G of FC (40G compressed with 4:1 compression) and 20G of IP
traffic. In this mode, only 10VE port mode is available. For information on configuring hybrid mode
and IP Extension features, refer to Configuration steps for IP Extension features on page 112.
In FCIP mode, you can configure the Brocade 7480 switch in either 10VE mode (default) or 20VE
mode using the extncfg --ve-mode -10VE|20VE command. This command is disruptive as it requires
rebooting the switch.
You can configure the following VE modes:
• 10VE mode: In this mode 10 of the 20 total VE_Ports on the switch are enabled. A single VE_Port
on a DP complex can use all Fibre Channel 20 Gbps bandwidth available to the DP complex. In
10VE mode, VE_Ports 29-33 and 39-43 are disabled.
• 20VE mode: This VE mode is available when the switch is in FCIP mode. In this mode, all 20
VE_Ports are enabled. A single VE_Port on a DP complex can use half of the available Fibre
Channel bandwidth available to the DP complex, a maximum of 10 Gbps. This option allows use of
more VE_ports, but at a lower maximum bandwidth.

NOTE
For the 7840 switch, only configure the maximum number of VE_Ports for the 7840 switch. 10VE
mode will accommodate nearly all environments and is the default.

NOTE
When switching modes, there can be no conflicting configurations or the extncfg command will fail.
For example, if you have a tunnel on VE30, you will not be allowed to switch to 10VE mode because
VE30 is disabled in that mode.

For more information on the Brocade 7840 switch port modes, refer to 10VE and 20VE port modes. on
page 52
Use the following steps to configure and display the Brocade 7840 switch operating modes for FCIP
mode.
1. Connect to the switch and log in using an account assigned to the admin role.
2. To set the switch to FCIP mode, enter the following:
extncfg --app-mode fcip
3. Perform one of the following steps:
• To set the operating mode to 20VE, enter the following:
switch:admin>extncfg --ve-mode 20VE
• To set the operating mode to 10VE, enter the following:
switch:admin>extncfg --ve-mode 10VE
4. To display the current operating mode, enter the following:
Switch:admin>extncfg--show
The following displays if the switch is in 20VE mode:
VE-Mode: configured for 20VE mode

78 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring port speed (Brocade 7840 switch)

Configuring port speed (Brocade 7840 switch)

You can configure the speed of 10 GbE ports on the Brocade 7840 switch to 1 Gbps or 10 Gbps
(default) using the portcfgge command.
Auto-negotiation is enabled by default (and forced enabled in 10G mode). When the port is set for 1G
mode, you can disable phy auto-negotiation using the portcfgge ge --disable -autoneg command.

NOTE
Auto-negotiation is for 1G GE Phy negotiation. It is not a speed negotiation. The GE port can be set to
either 1G mode or 10G mode. A port set in autonegotiate mode is negotiating full duplex and pause
frames (802.3X) with the attached switch. The port will not come up if there is an autonegotiate
mismatch with the attached switch.

Use the following steps to configure port speed on the Brocade 7840 switch 10 GbE ports:
1. Connect to the switch and log in using an account assigned to the admin role.
2. Perform one of the following steps:
• To set the port speed at 1 Gbps for port ge4, enter the following:
switch:admin>portCfgGe ge4 --set -speed 1G
• To set the port speed at 10 Gbps for port ge4, enter the following:
switch:admin>portCfgGe ge4 --set -speed 10G
• To disable autonegotiate on port ge4, enter the following:
switch:admin>portCfgGe ge4 --disable -autoneg
• To enable autonegotiate on port ge4, enter the following:
switch:admin>portCfgGe ge4 --enable -autoneg
3. To display current port speed configuration for ge4, enter the following:
switch:admin>portCfgGe ge4 --show

Configuring an IPIF
You must configure an IP interface (IPIF) for each circuit that you intend to configure on a Ethernet port.
This is done using the portCfg ipif create command. The IP interface consists of an IP address,
netmask, an IP MTU size, and other options depending on the extension switch or blade.
The following examples create the addressing needed for the basic sample configuration in the below
figure using a Brocade 7800 switch and FX8-24 blade.
The following command creates an IP interface for port ge0 on the Brocade FX8-24 blade in slot 8 of
the Brocade DCX-4S.
switch:admin> portcfg ipif 8/ge0 create 192.168.1.24 netmask 255.255.255.0 mtu 1500

The following command creates an IP interface for port ge0 on the Brocade 7800 switch.
switch:admin> portcfg ipif ge0 create 192.168.1.78 netmask 255.255.255.0 mtu 1500

The following command displays current configuration details for all interfaces.
switch:admin> portshow ipif

Brocade Fabric OS Extension Administrator's Guide 79

53-1003507-05
Configuring an IP route

FIGURE 16 Basic sample configuration

Requirements and options for configuring ipifs include the following:

• There are no addressing restrictions for IPv4 and IPv6 connections with both switches or blades in
the tunnel running Fabric OS v7.0 and later.
• You can use CIDR notation for the IP4 addresses like you can for IPv6 addresses.
• You can specify an optional IP MTU size. If not specified, the size will be set to 1500 bytes.
The Brocade 7840 switch has the following additional requirements and options for configuring an ipif:
• You must assign a DP complex to the Ethernet port where commands will be received.
• For MTU the auto option can be used instead of a value which will cause any circuits using this IP
address to use PTMU discovery to set the desired IP MTU.
• You can specify a VLAN ID (optional). Note that this is the only method to set the VLAN for the IP
address.
The following command creates an IP interface for port ge0 on a Brocade 7840 switch. Because the
Brocade 7840 switch DP0 and DP1 share Ethernet ports, port ge0 is assigned to DP0 so that DP0
receives the command. Note that a network mask (netmask), VLAN ID (vlan), and IP MTU (mtu) are
specified. If CIDR is used, a prefix (pfx) is used instead of the netmask.
switch:admin>portcfg ipif ge0.dp0 create 192.168.0.10/24 vlan 100 mtu 1400

NOTE
For full details on syntax and using the portcfg ipif and portshow ipif commands, refer to the Fabric
OS Command Reference.

Configuring an IP route
Routing is based on the destination IP address presented by an extension circuit. If the destination
address is not on the same subnet as the Ethernet port IP address, you must configure an IP route to
that destination with an IP gateway on the same subnet as the local Ethernet port IP address.
You can define up to 32 routes for each GbE port on the Brocade 7800 switch and FX8-24 blade. You
can define up 128 routes per GbE port on the Brocade 7840 switch; however, you can only define 120
routes per DP. Note that the portshow iproute command can display more routes than those you
configured once all routes are added.

80 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Commands for configuring IP routes

To configure a route, use the portCfg iproute create command to specify the destination IP address,
subnet mask, and address for the gateway router that can route packets to the destination address.
Optionally, on the Brocade FX8-24 blade, you can configure an IP route for a failover crossport using
the -x or - - crossport option. For information on configuring IP routes using crossport addresses, refer
to Configuring IP routes with crossports on page 42.
The following figure shows an IP route sample configuration.

FIGURE 17 Configuring an IP route

Commands for configuring IP routes

Following are examples of commands for configuring IP routes. You can use the same commands for
Brocade FX8-24 blades, 7800 switches, and 7840 switches with modifications as noted.
The following commands are used to configure an IP route for the example configuration in the previous
illustration.
• The following command creates an IP route to destination network 192.168.11.0 for port ge0 on the
FX8-24 blade in slot 8 of the Brocade DCX-4S Backbone. The route is through local gateway
192.168.1.1. After the destination address, either specify a pfx (prefix length) or network mask.
switch:admin> portcfg iproute 8/ge0 create 192.168.11.0 netmask 255.255.255.0
192.168.1.1
• The following command creates an IP route to destination network 192.168.1.0 for port ge0 on the
Brocade 7800 switch. The route is through local gateway 192.168.11.1. After the destination address,
either specify a pfx (prefix length) or network mask.
switch:admin> portcfg iproute ge0 create 192.168.1.0 netmask 255.255.255.0
192.168.11.1
• The following command displays configured IP route information for port ge0.
switch:admin> portshow iproute ge0

The following command creates an IP route to destination network 192.168.12.100 for port ge0 on a
Brocade 7840 switch. The route is through local gateway 192.168.1.1. Because Ethernet ports are
shared between DP complexes, the ge1.dp0 option directs the command to a specific DP.
portcfg iproute ge1.dp0 create 192.168.12.100 netmask 255.255.255.255 192.168.1.1

NOTE
For additional IP route configuration examples and related output from portshow iproute commands,
refer to the Fabric OS Command Reference.

Brocade Fabric OS Extension Administrator's Guide 81

53-1003507-05
Commands for modifying IP routes

Commands for modifying IP routes

On the Brocade 7840, you can modify an existing IP route to change the local gateway address of an
IP route using the portcfg iproute port modify command. You cannot use this command to modify the
destination network address. If this needs to be modified, you must delete the IP route, and then
recreate it. Also, you cannot use this command to change the prefix length or network mask.
Following is a command to change the local gateway address on a Brocade 7840 switch to
192.168.11.1. Note that for the Brocade 7840 switch only, a DP destination (in this case, dp0) is
specified for the Ethernet port. Because Ethernet ports are shared between DP complexes, this option
directs the command to a specific DP.
portcfg iproute ge1.dp0 modify 192.168.12.100 netmask 255.255.255.255 192.168.11.1

Validating IP connectivity
The following example tests the connectivity between the FX8-24 blade and the Brocade 7800 switch
in the basic sample configuration from the Brocade 7800 switch. The -s option specifies the source
address, and the -d option specifies the destination address.
switch:admin> portcmd --ping ge0 -s 192.168.11.78 -d 192.168.1.24

When using VLANS, VLAN tagging ensures that test traffic traverses the same path as real traffic. A
VLAN tag entry for both the local and remote sides of the route must exist prior to issuing the
portCmd --ping command. Refer to Managing the VLAN tag table on page 105 for details.

NOTE
To ping crossport addresses, refer to Using ping with crossports on page 44.

Creating an Extension tunnel

Create tunnels using the portCfg fciptunnel create command. You configure specific tunnel options
using this command, such as compression, IPsec, and FICON emulation or acceleration options. You
can also specify local and remote IP addresses and circuit parameters for default circuit 0.

NOTE
When circuit options are specified on the portcfg fciptunnel create command and the portcfg
fciptunnel modify command, they apply only to circuit 0. When additional circuits are added, circuit
options must be applied per circuit using the portcfg fcipcircuit create or the portcfg fcipcircuit
modify command.
A suggested technique is to configure the tunnel with appropriate tunnel parameters only (no IP
addresses or circuit options). This may be useful in staging a configuration without committing specific
circuit parameters. Then you can configure circuit 0 and additional circuits using portcfg fcipcircuit
commands.

82 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Creating an FX8-24 and Brocade 7800 tunnel

NOTE
A Brocade 7840 switch can only connect with another Brocade 7840 switch through an extension
tunnel. It cannot connect to a Brocade 7500 switch, 7800 switch, or FX8-24 blade.

Creating an FX8-24 and Brocade 7800 tunnel

NOTE
You cannot create a tunnel from a Brocade 7840 switch to a Brocade 7800 switch or FX8-24 blade.
To create an FX8-24 tunnel endpoint using the portcfg fciptunnel command, VE_Port 12 is specified
on slot 8. Circuit 0 is created automatically when the tunnel is created. A tunnel that carries FCIP traffic
is represented by a VE_Port. The Brocade 7800 switch remote or destination address (192.168.11.78)
is specified first, followed by the FX8-24 local or source address (192.168.1.24). ARL minimum (-b) and
maximum (-B) committed rates are specified for circuit 0.
switch:admin> portcfg fciptunnel 8/12 create --remote-ip 192.168.11.78 --local-ip
192.168.1.24 -b 500000 -B 1000000

The following command creates the Brocade 7800 tunnel endpoint. VE_Port 16 is specified. Circuit
parameters are included to create circuit 0 on the Brocade 7800 switch. The circuit parameters must
match up correctly with the circuit parameters on the FX8-24 end of the circuit. The FX8-24 remote or
destination address is specified first (192.168.1.24), followed by the 7800 switch local or source address
(192.168.11.78). Matching ARL minimum and maximum committed rates must be specified on both
ends of circuit 0.
switch:admin> portcfg fciptunnel 16 create --remote-ip 192.168.1.24 --local-ip
192.168.11.78 -b 500000 -B 1000000

For a description of circuit and tunnel configuration options that you can include on the portcfg
fciptunnel command, refer to Tunnel configuration options on page 84.
The following figure illustrates the results of the configuration.

FIGURE 18 Adding a tunnel to the basic sample configuration

Brocade Fabric OS Extension Administrator's Guide 83

53-1003507-05
Creating Brocade 7840 tunnels

Creating Brocade 7840 tunnels

Tunnel configuration for Brocade 7840 switches is similar to Brocade 7800 and FX8-24 switches, but
have some different tunnel options.
You can only create a tunnel from a Brocade 7840 switch to another Brocade 7840 switch. You cannot
create tunnels between Brocade 7840 switches and Brocade 7500 switches, 7800 switches, or
FX8-24 blades.
The following example configures a tunnel endpoint on a Brocade 7840 switch at VE_Port 25.

switch:admin> portcfg fciptunnel 25 create --local-ip 192.168.2.15 --remote-ip

192.168.2.25 -b 500000 -B 10000000 -c deflate --ipsec policy1

• To create a Brocade 7840 switch endpoint using the portcfg fciptunnel command, VE_Port 25 is
specified.
• The circuit 0 local or source address (192.168.2.15) is specified, followed by the remote or
destination address (192.168.2.25).
• ARL minimum (-b) and maximum (-B) committed rates are specified for circuit 0. These must match
the rates configured on the remote 7840 switch.
• Deflate compression is enabled.
• IPsec is enabled using policy1
The following command creates the destination Brocade 7840 switch endpoint. VE_Port 34 is
specified.
switch:admin> portcfg fciptunnel 34 create --local-ip 192.168.2.25 --remote-ip
192.168.2.15 -b 500000 -B 1000000 -c deflate --ipsec policy1

For the destination switch, the same circuit and tunnel options are configured as the other Brocade
7840 switch. The circuit options for circuit 0, such as ARL minimum and maximum committed rates,
must match.
For a description of circuit and tunnel configuration options that you can include on the portcfg
fciptunnel command, refer to Tunnel configuration options on page 84.

Tunnel configuration options

Most Extension features for tunnels that carry FCIP traffic are enabled using optional arguments
available on the portcfg fciptunnel create command and the portcfg fciptunnel modify command.
Some of these arguments apply only to tunnels, and are used only on the portcfg fciptunnel create
command and the portcfg fciptunnel modify commands.
For information on the details of tunnel and circuit configuration options available on the portcfg
fciptunnel command, refer to Fabric OS Command Reference.

Keep-alive timeout option

Consider the following items when configuring the keep-alive timeout value (KATOV):
• A tunnel that carries FICON traffic requires a KATOV of less than or equal to 1 second for each
circuit added to a tunnel.
• If the tunnel is created first with the FICON flag, then the KATOV for all added circuits will be 1
second (recommended value for FICON configurations).
• If the tunnel is created with one or more circuits, and the tunnel is modified to be a FICON tunnel,
then the circuits that were previously created must be modified to have the correct KATOV.

84 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Creating additional circuits

• Set the circuit KATOV to the same value on both ends. If local and remote circuit configurations do
not match, the tunnel will use the lower of the configured values.
• For normal extension tunnel operations over tunnels transporting FICON traffic, the KATOV for all
circuit members of a VE_Port (tunnel) must be less than the overall I/O timeout for all FC exchanges.
If the FC I/O timeout value is less than the KATOV, then inputs and outputs will time out over all
available circuits without being retried.
• The default timeout values are not the same for every supported platform. The Brocade FX8-24
value is 10 seconds. The value for the Brocade 7840 is 6 seconds. The default value will also differ
depending on whether FICON emulation is enabled.
The KATOV should be based on application requirements. Check with your FC initiator providers to
determine the appropriate KATOV for your application. The sum of KATOVs for all circuits in a tunnel
should be close to the overall FC initiator I/O timeout value. As an example, a mirroring application has
a 6-second I/O timeout. There are three circuits belonging to the VE_Port (3 circuit members in the
tunnel). Set the KATOV to 2 seconds on each circuit. This will allow for maximum retries over all
available circuits before an I/O is timed out by the initiator.
Refer to the keep-alive timeout option in Tunnel configuration options on page 84 for information on
option format and value range.

Creating additional circuits

Additional circuits can be created and added to a tunnel (VE_Port) using the portCfg fcipcircuit create
command. The following examples add a circuit to the tunnel in the basic sample configuration
illustrated in Creating an Extension tunnel on page 82. Note that although these examples use a
FX8-24 blade and a Brocade 7800 switch, examples for the Brocade 7840 switch would be similar.

NOTE
For the 7800 switch and FX8-24 blade, you must enable the Advanced Extension (FTR_AE) license to
add circuits. This license is enabled on the 7840 switch when shipped from the factory.
The following command creates circuit 1 on the FX8-24 end of the trunk.
switch:admin> portcfg fcipcircuit 8/12 create 1 --remote-ip 192.168.11.78 --local-ip
192.168.1.25 -b 15500 -B 62000

The following command displays configuration details for circuit 1.

switch:admin> portshow fcipcircuit 8/12 1

The following command creates circuit 1 on the Brocade 7800 switch end of the trunk.
switch:admin> portcfg fcipcircuit 16 create 1 --remote-ip 192.168.1.25 --local-ip
192.168.11.78 -b 15500 -B 62000

The following command displays configuration details for circuit 1.

switch:admin> portshow fcipcircuit 1

Note the following information about the basic sample configuration:

• The VE_Ports used to create the tunnel are the same as those specified on the tunnel in the basic
sample configuration. The VE_Ports uniquely identify the trunk, and the circuit is associated with this
specific trunk.
• The unique destination and source IP addresses are mirrored on either end of the tunnel. The
address 192.168.11.78 is the destination address for the FX8-24 blade, and the source address for
the Brocade 7800 switch, while the address 192.168.1.25 is the destination address for the Brocade
7800 switch, and the source address for the FX8-24 blade.

Brocade Fabric OS Extension Administrator's Guide 85

53-1003507-05
Verifying the tunnel configuration

Also note the following about configuring circuits in general:

• ARL minimum and maximum rates are set per circuit. They must be the same on either end of a
circuit, but individual circuits may have different rates.
• You can configure standby circuits to operate during circuit failover by assigning a metric. In the
following example, circuit 2 is used only when circuit 1 fails.
switch:admin> portcfg fcipcircuit 8/12 create 1 --remote-ip 192.168.11.78 --local-
ip 192.168.1.25 -b 155000 -B 620000
switch:admin> portcfg fcipcircuit 8/12 create 2 --remote-ip 192.168.11.8 --local-
ip 192.168.1.26 -b 155000 -B 620000 -x 1
• When multiple tunnels are present on a switch and additional circuits are added to an active tunnel,
some frame loss can occur for a short period of time because the internal Fibre Channel frame
routing tables in the switch are refreshing. Therefore, add additional circuits only during low I/O
periods on the VE_Port trunk being modified. In addition, if deleting or adding a circuit increases or
decreases the total trunk bandwidth, then disable and re-enable the tunnel (VE_Port) after deleting
or adding the circuit. This will allow the switch to adjust internal routes to fully utilize the new
bandwidth. Tunnels and trunks are designed to produce FSPF costs that make them unlikely to be
the preferred path to reach all destinations within the local site. This prevents local traffic from
taking a long path across the connection.

Verifying the tunnel configuration

After you have created local and remote configurations, verify that the tunnel and circuit parameters
are correct using the portshow fciptunnel command. Refer to the Fabric OS Command Reference
for a description of the command syntax and output.

Configuring Extension HCL

Extension HCL is an optional configuration for the Brocade 7840 Extension Switch. The only
configuration that you must provide for Extension HCL is additional IP addresses for the local backup
tunnel (LBT) and remote backup tunnel (RBT) endpoints. There are no subnet restrictions for
configuring the IP addresses for these endpoints. The IP network must be capable of delivering traffic
to all these IP address from any IP address. The tunnel and circuit parameters of the main tunnel (MT)
or trunk are automatically replicated on the LBT and RBT, including the circuit properties such as QoS
markings, FastWrite, and FICON Acceleration. This maintains the environment during the Extension
HCL process. For more information on the Extension HCL feature, refer to Extension Hot Code Load
on page 53.

NOTE
Hot code load (HCL) is disruptive when the Brocade 7840 is in hybrid mode. IP traffic will be
interrupted during HCL.
A LBT that is protecting an MT will stay online during the Extension HCL process. The Brocade 7840
can be configured to operate either in 10VE or 20VE mode for Extension HCL. Each DP on 10VE
mode can accommodate 5 MTs and 5 LBTs. If 5 protected tunnels were configured, there would be a
maximum of 15 tunnels (5 MTs, 5 LBTs, and 5 RBTs). Each DP in 20VE mode can accommodate 10
MTs and 10 LBTs. If 10 protected tunnels were configured, there would be a maximum of 30 tunnels
(10 MTs, 10 LBTs, and 10 RBTs).

86 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 DP and tunnel configuration

An MT must contain at least one protected circuit. To configure a protected circuit, you must provide two
additional IP addresses for the circuit. Not all circuits must be protected by Extension HCL. Only the
circuits that are configured with the additional Extension HCL IP addresses will stay up during firmware
updates. The non-protected circuits will go down. The trunk would run at a diminished capacity if there
were not a one-for-one protection of the circuits resulting in a lesser aggregate bandwidth.
To provide the additional IP addresses, use the --local-ha-ip and --remote-ha-ip options in the portcfg
fcipcircuit create command. You can also use these options in the portcfg fciptunnel create
command if you wish to create circuit 0 using that command. Following is an example of creating a
protected circuit for a tunnel.
portcfg fcipcircuit <ve_port> create <circuit_ID> --local-ip <ipaddr> --remote-ip
<ipaddr> --local-ha-ip <ipaddr> --remote-ha-ip <ipaddr>

For detailed instructions to create backup tunnels for the Extension HCL feature, refer to Configuring
backup tunnels on page 89.
Once you configure backup tunnels, you can monitor Extension HCL status on tunnels during the
firmware download using the portshow fciptunnel --hcl-status command as in the following example.
switch:admin> portshow fciptunnel –-hcl-status
Checking FCIP Tunnel HA Status.

Current Status : Ready

CP Version : v7.4.0
DP0 Status:
State : Online - Inactive
Version : v7.4.06
Current HA Stage : INITIAL
DP1 Status:
State : Online - Inactive
Version : v7.4.0
Current HA Stage : INITIAL

Tunnel 24 HA configure but HA Offline. Traffic will be disrupted.

DP and tunnel configuration

This information provides considerations for configuring DPs, main, local, and remote tunnels with
Extension HCL.
When you configure interfaces and tunnels for Extension HCL, the main tunnel (MT) interface cannot be
on the same data processor (DP) complex as the local backup tunnel (LBT). Because the Brocade 7840
switch shares Ethernet ports, you can assign an Ethernet port to a specific DP using the portcfg ipif
command with the ge_port.dp_num option.
In the following figure, the 7840-A switch uses DP0 as its home for the MT. DP1 is used for the LBT.
Assuming you define a single Extension HCL one-circuit tunnel on VE 24 between 7840-A and 7840-B,
and both switches use VE 24 and ge2, the configuration of IPIFs is completed on a per DP basis.

FIGURE 19 Extension HCL configuration with VT, LBT, and RBT

The DP complex creates the following TCP connections between 7840-A and 7840-B:

Brocade Fabric OS Extension Administrator's Guide 87

53-1003507-05
Configuring Extension Features

• 192.168.2.15 to 192.168.11.78—the main TCP connections for the MT between the switches
• 192.168.2.15 to 192.168.11.68—the connection between 7840-A RBT and 7840-B LBT
• 192.168.2.31 to 192.168.11.78—the connection between 7840-B RBT and 7840-A LBT
The following table shows the possible options for configuring MTs and LBTs. The VE ports used on
the local and remote switches (7840-A and 7840-B in the example) need not be bidirectional. That is,
you can use VE_Port 24-33 on the local switch and VE_Port 34-43 on the remote switch for the MT.

TABLE 14 Extension HCL considerations for VE port configuration

7840-A Switch VE PT local IP LBT local IP 7840-B Switch VE RBT IP addresses

Ports addresses defined addresses defined Ports defined on
on on

24–33 One of ge0-ge17 on One of ge0-ge17 on 24–33 One of ge0-ge17 on

dp0 dp1 dp1
(such as ge0.dp0) (such as ge0.dp1) (such as ge0.dp1)

24–33 One of ge0-ge17 on One of ge0-ge17 on 34–43 One of ge0-ge17 on

dp0 dp1 dp0
(such as ge0.dp0) (such as ge0.dp1) (such as ge0.dp0)

34–43 One of ge0-ge17 on One of ge0-ge17 on 34–43 One of ge0-ge17 on

dp1 dp0 dp0
(such as ge0.dp1) (such as ge0.dp0) (such as ge0.dp0)

34–43 One of ge0-ge17 on One of ge0-ge17 on 24–33 One of ge0-ge17 on

dp1 dp0 dp1
(such as ge0.dp1) (such as ge0.dp0) (such as ge0.dp1)

DP and tunnel configuration example

The following example shows the configuration commands for the 7840-A switch.

portcfg ipif ge2.dp0 create 192.168.2.15/24 mtu 9100

portcfg ipif ge2.dp1 create 192.168.2.31/24 mtu 9100

/* Note:
portcfg iproute commands excluded – but would be required as the IPIFs are on
different networks
*/
portcfg fciptunnel 24 create –f –t -F -c fast-deflate --ficon-xrc --ficon-tape
/* create an empty tunnel with FastWrite/OSTP/HW Compression and FICON Features */

portcfg fcipcircuit 25 create 0 --local-ip 192.168.2.15 --remote-ip 192.168.11.78 --

local-ha-ip 192.168.2.31 --remote-ha-ip 192.168.11.68 -b 500000 -B 1000000

88 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring backup tunnels

Configuring backup tunnels

A backup tunnel is required for each DP on a Brocade 7840 switch to support the Extension hot code
load (HCL) feature. Configure these tunnels, along with the MT, using the portcfg fciptunnel and
portcfg fcipcircuit commands.
An example for creating local and remote IP addresses using the portCfg fcipcircuit create command
follows:
portcfg fcipcircuit 24 create 1 --local-ip 192.168.2.15 --remote-ip
192.168.11.78 --local-ha-ip 192.168.2.31 --remote-ha-ip 192.168.11.68 -b 155000 -B
620000

The --local-ip and --remote-ip options provide the MT IP addresses. The --local-ha-ip and --remote-ha-ip
options provide the LBT and RBT IP addresses respectively.
The following steps illustrate how to configure two circuits (0 and 1) for ge2, each with Extension HCL
IP addresses, for a tunnel from source VE_Port 24.
1. Configure IPIFs that you will use for circuit 0 as in the following example.

NOTE
The local interface (Ethernet port) assigned to the LBT IP address must be assigned to the DP other
than the DP where the local MT IP address is assigned. Because the Brocade 7840 switch shares
Ethernet ports, you can assign an Ethernet port to a specific DP using the portcfg ipif command with
the ge_port.dp_num option.
switch:admin> portcfg ipif ge2.dp0 create 192.168.2.10 netmask 255.255.255.0 mtu
1500
switch:admin> portcfg ipif ge2.dp1 create 192.168.2.30 netmask 255.255.255.0 mtu
1500
The ge2.dp0 option assigns port ge2 to DP0 and ge2.dp1 assigns ge2 to DP1. The address assigned
to ge2 on DP0 will be for the local MT, while the address assigned to ge2 on DP1 will be the LBT.
2. Configure ipifs that you will use for circuit 1 in the same fashion, as in the following example:
switch:admin> portcfg ipif ge2.dp0 create 192.168.2.15 netmask 255.255.255.0 mtu
1500
switch:admin> portcfg ipif ge2.dp1 create 192.168.2.31 netmask 255.255.255.0 mtu
1500
3. Configure a tunnel for VE_24 and circuit 0 for the tunnel using the following example.
switch:admin> portcfg fciptunnel 24 create --local-ip 192.168.2.10 --remote-ip
192.168.2.20 --local-ha-ip 192.168.2.30 --remote-ha-ip 192.168.2.40 -b 12000 -B
1000000
4. Configure circuit 1 for the tunnel using the following example:
switch:admin> portcfg fcipcircuit 24 create 1 --local-ip 192.168.2.15 --remote-ip
192.168.2.25 --local-ha-ip 192.168.2.31 --remote-ha-ip 192.168.2.41 -b 15500 -B
62000
5. Verify the tunnel configuration using the portshow fciptunnel --ha --circuit command as in the
following example:
switch:admin> portshow fciptunnel --ha --circuit

Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met/G
----------------------------------------------------------------------------
24 - Up -M-fTF-a 2h21m40s 21.03 18.52 1 - -/-
24 0 ge2 Up ---rh--4 2h21m40s 10.51 9.27 1 4000/5000 0/-
24 1 ge2 Up ---rh--4 2h21m36s 10.77 9.25 1 4000/5000 0/-
25 - Up -R-fTF-a 2h21m39s 0.00 0.00 1 - -/-
25 0 ge2 Up ---rh--4 2h21m40s 0.00 0.00 1 4000/5000 0/-
25 1 ge2 Up ---rh--4 2h21m37s 0.00 0.00 1 4000/5000 0/-
25 - Up -L-fTF-a 2h21m43s 0.00 0.00 1 - -/-
25 0 ge2 Up ---rh--4 2h21m43s 0.00 0.00 1 4000/5000 0/-
25 1 ge2 Up ---rh--4 2h21m41s 0.00 0.00 1 4000/5000 0/-
----------------------------------------------------------------------------
Flags (tunnel): M=MainTunnel L=LocalBackup R=RemoteBackup
i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
A=AdvCompr a=FastDeflate d=Deflate D=AggrDeflate

Brocade Fabric OS Extension Administrator's Guide 89

53-1003507-05
Enabling persistently disabled ports

(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU 4=IPv4 6=IPv6

ARL a=Auto r=Reset s=StepDown t=TimedStepDown

Enabling persistently disabled ports

Once the tunnel configurations are complete, the VE ports must be persistently enabled.
Use the following steps to enable persistently disabled ports.
1. Connect to the switch and log in using an account assigned to the admin role.
2. Enter the portCfgShow command to view ports that are persistently disabled.
3. After identifying the ports, enter the portCfgPersistentEnable command to enable the ports.
4. Enter the portCfgShow command to verify the port is persistently enabled.

Disabling ports with FMS Mode enabled

If you enter portCfgPersistentEnable and receive "command not allowed in fmsmode" or "command
not found" messages, FMS mode may be enabled. You cannot use the portCfgPersistentEnable or
portCfgPersistentDisable commands with FMS mode enabled. Use the portEnable and
portDisable commands instead.
You can determine if FMS mode is enabled by using the ficoncupshow fmsmode command.

NOTE
The portCfgPersistence command also cannot be used with FMS mode enabled. This command
sets or removes the persistent disable flag on a port or range of ports. If FMS mode is enabled, use
the portdisable command with Active=Saved mode enabled instead.

Modifying a tunnel
Tunnel characteristics and options can be modified as needed using the portCfg fcipTunnel
command with the modify option. The command syntax is as follows:
portCfg fciptunnel ve_port modify options
The VE_Port variable indicates the specific VE_Port to which each tunnel is assigned. The VE_Port
number serves as the tunnel ID. The range is 16 through 23 for a Brocade 7800 switch and 12 through
31 for the FX8-24 blade.
The options variable indicates the choice of options. Refer to Fabric OS Command Reference for
information about the portCfg fciptunnel command.

NOTE
When you use portcfg fciptunnel to modify the circuit options, the changes apply only to circuit 0.

CAUTION

Using the modify option may disrupt traffic on the specified tunnel for a brief period of time.

90 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Modifying a circuit

Modifying a circuit
Circuit characteristics and options can be modified as needed using the portCfg fcipcircuit command
with the modify option. The general command syntax is as follows:
portCfg fcipcircuit ve_port modify circuit_id options

ve_port Each tunnel or trunk is assigned to a specific VE_Port. The VE_Port number serves as the
tunnel ID. Specify the VE_Port of the tunnel that contains the circuit you want to modify. The
range for VE_Ports varies for extension switches and blades. Refer to the section for your
switch or blade in Tunnel and circuit requirements on page 61 for information.

circuit_id The numeric ID assigned when the circuit was created.

options Refer to the Fabric OS Command Reference for descriptions of the options.

NOTE
You can modify all circuits, including circuit 0, using the portCfg fcipcircuit command.
For full details on syntax and using this command, refer to the Fabric OS Command Reference.

Deleting an IP interface
You can delete an IP interface using the portcfg ipif command with the delete option. The command
syntax is as follows:
portcfg ipif [slot/]ge n delete ipaddr
For full details on syntax and using this command, refer to the Fabric OS Command Reference.

NOTE
You cannot delete an IP interface if there is a tunnel or circuit configured to use it. Be sure to delete all
tunnels, circuits, and IP routes using an interface before deleting it.

Deleting an IP route
You can delete an IP route to a gateway destination IP address using the portcfg iproute command
with the delete option. The command syntax is as follows for both IPv4 and IPv6 addressing:
portcfg iproute [slot/]ge n delete dest_ipv4 netmask mask
portcfg iproute [slot/]ge n delete dest_ipv6/prefix_len
For full details on syntax and using this command, refer to the Fabric OS Command Reference.

NOTE
You cannot delete an IP route if there is a tunnel, circuit, or IP interface configured to use it. Be sure to
delete all tunnels and circuits using an IP route before deleting the IP route.

Brocade Fabric OS Extension Administrator's Guide 91

53-1003507-05
Deleting a trunk

Deleting a trunk
When you delete a trunk, you also delete all associated circuits. Use the portCfg fciptunnel
command with the delete option to delete FCIP tunnels. The command syntax is as follows:
portcfg fciptunnel ve_port delete
For full details on syntax and using this command, refer to the Fabric OS Command Reference.

CAUTION

The fciptunnel delete command does not prompt you to verify your deletion. Be sure you want
to delete the tunnel before you press Enter.

NOTE
You must delete a tunnel before you can delete an IP route that it uses and the IP interface that uses
the route.

Deleting a circuit
You can delete individual circuits using the portCfg fcipcircuit command with the delete option. The
command syntax is as follows:
portcfg fcipcircuit ve_port delete circuit_id
For full details on syntax and using this command, refer to the Fabric OS Command Reference.

Configuring Per-Priority TCP QoS priorities over a trunk

Per-Priority TCP QoS (PP-TCP-QoS) prioritizes FC traffic flows between initiators and targets within a
tunnel to optimize bandwidth and performance.
Each circuit has multiple TCP connections that manage traffic over a tunnel, as illustrated in the figure
below. Each circuit handles one of the following priority traffic types.
• F class—F class is the highest priority, and is assigned bandwidth as needed at the expense of
lower priorities, if necessary. This is referred to as strict priority.
• QoS high - The default priority value is 50 percent of the available bandwidth.
• QoS medium - The default value is 30 percent of the available bandwidth.
• QoS low - The default value is 20 percent of the available bandwidth.
QoS high, medium, and low priority traffic are assigned a percentage of available bandwidth based on
priority level. QoS priority is based on the Virtual Circuit (VC) that carries data into the DP complex.
For example, if data enters on a high VC, it is placed on a high TCP connection; if it enters on a low
VC, then it is placed on the low TCP circuit. Data is assigned to the proper VC based on zone name
prefix.
The following figure illustrates the internal architecture of TCP connections that handle PP-TCP-QoS.
Note that this illustrates a tunnel containing a single circuit only.

92 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Modifying default priority values

FIGURE 20 TCP connections for handling QoS

Modifying default priority values

You can modify the default QoS priority values on Brocade extension switches and blades. Note that
this only changes the QoS priority distribution in the tunnel and does not reconfigure the fabric.
Change the priority percentages on 8 Gbps extension platforms using the optional percentage tunnel
argument for the portcfg fciptunnel create and portcfg fciptunnel modify commands. When
configuring QoS percentages for each level, remember the following:
• The three values must equal 100 percent.
• A minimum of 10 percent is required for each level.
• QoS priority settings must be the same on each end of the tunnel.

Brocade Fabric OS Extension Administrator's Guide 93

53-1003507-05
Using logical switches

NOTE
Priorities are enforced only when there is congestion on the network. If there is no congestion, all
traffic is handled at the same priority.
The following command sets the QoS priority ratios on VE_Port 12 to high (50%), medium (40%) and
low (10%) priorities respectively.
portcfg fciptunnel 1/12 create --qos-bw-ratio 50,40,10

The following command displays details of the tunnel configuration, including set QoS percentages.

portshow fciptunnel 1/12

For more information on using Fabric OS commands, optional arguments, and command output, refer
to the Fabric OS Command Reference.

Using logical switches

Configuring tunnels and other components in switches enabled for Virtual Fabrics is somewhat
different than on switches not enabled for Virtual Fabrics. This section provides a brief overview of
common logical switch concepts and terminology followed by the specifics of configuring logical
switches.

Logical switch overview

The logical switch feature allows you to divide a physical chassis into multiple fabric elements. Each of
these fabric elements is referred to as a logical switch. Each logical switch functions as an
independent self-contained FC switch. Each chassis can have multiple logical switches.
Logical switches are used to take advantage of multiple VE_Ports. Emulation with ECMP (multiple
same-cost VE_Ports) to the same domain are not supported in all emulation modes.

Default logical switch

Virtual Fabrics allows Ethernet ports in the default switch to be shared among VE_Ports in any logical
switch. To use the Virtual Fabrics features, you must first enable Virtual Fabrics on the switch.
Enabling Virtual Fabrics creates a single logical switch in the physical chassis. This logical switch is
called the default logical switch, and it initially contains all of the ports in the physical chassis. After you
enable Virtual Fabrics, you can create additional logical switches. The number of logical switches that
you can create depends on the switch model.
After you create logical switches, the chassis appears as multiple independent logical switches. All of
the ports continue to belong to the default logical switch until you explicitly move them to other logical
switches. The default logical switch always exists. You can add and delete other logical switches, but
you cannot delete the default logical switch unless you disable Virtual Fabrics.

Creating logical switches

To create logical switches and logical fabrics, you must perform the following steps.

94 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Port assignment

1. Enable Virtual Fabrics mode on the switch using instructions in the "Managing Virtual Fabrics"
chapter of the Fabric OS Administrator's Guide.
2. Configure logical switches to use basic configuration values using instructions in the "Managing
Virtual Fabrics" chapter of the Fabric OS Administrator's Guide.
3. Create logical switches using instructions for creating a logical switch or base switch in the
"Managing Virtual Fabrics" chapter of the Fabric OS Administrator's Guide.

Port assignment
Initially, all ports belong to the default logical switch. When you create additional logical switches, they
are empty and you can assign ports to those logical switches. As you assign ports to a logical switch,
the ports are moved from the default logical switch to the newly created logical switch. Following are
some requirements for assigning ports:
• A given port can be in only one logical switch.
• You can move ports from one logical switch to another.
• A logical switch can have as many ports as are available in the chassis.
• Ports with defined configuration settings in a logical switch or the default switch cannot be moved to
another logical switch without first deleting the current settings. For example, you cannot move a VE_
Port with a defined tunnel in the default switch or a logical switch to a different logical switch until you
delete the circuits and the tunnel in the logical switch currently containing the port that you want to
move. Similarly, you cannot move a GE_Port between logical switches until all IP routes and IP
interfaces have been deleted in the logical switch currently containing the port that you want to move.
Use the lsCfg --config slot/ge _port command to move ports from one logical switch to a different logical
switch. The FID is the fabric ID of the logical switch where you want to move the ports. The ports are
automatically removed from the logical switch where they are currently assigned.
As a recommended best practice, leave Ethernet interfaces in the default logical switch and do not
move them to another logical switch. There is no reason to move them because of the Ethernet Port
Sharing (EPS) feature. A VE_Port in any logical switch context can use an Ethernet interface in the
default switch. In addition, by moving a physical port from the default switch to a logical switch, it will not
be available to tunnels configured in other logical switches. Refer to Ethernet Port sharing on page 96
for details.

Logical switches and fabric IDs

When you create a logical switch, you must assign it a fabric ID (FID). The fabric ID uniquely identifies
the logical switch within a chassis and indicates the fabric to which the logical switch belongs. You
cannot define multiple logical switches with the same fabric ID within the chassis. A logical switch in one
chassis can communicate with a logical switch in another chassis (or to a switch not enabled for logical
switches) only if the switches have the same fabric ID (FID). The default logical switch is initially
assigned FID 128, which can be changed.
Only logical switches with the same FID can form a logical fabric. If you connect two logical switches
with different FIDs, the link between the switches segments.
Create logical switches using the lsCfg command. For details, refer to the instructions for creating a
logical switch or base switch section in the Fabric OS Administrator's Guide and to the lsCfg command
in the Fabric OS Command Reference.

Logical switch contexts

You can configure features or perform other tasks on a specific logical switch as you would any Fibre
Channel switch by entering commands while in the "context" of that logical switch, which is the FID of

Brocade Fabric OS Extension Administrator's Guide 95

53-1003507-05
Connecting logical switches

the logical switch. Note that "128" is sometimes referred to the context for the default switch as that is
the initial FID of the default switch when you enable Virtual Fabrics. However, this FID may be
changed.
There are two methods for changing to the context of a specific logical switch so that you can perform
configuration or other tasks:
• Use the setcontext fabricID command. This changes the context to a specific logical switch and
changes the command line prompt to reflect the new FID. Any commands entered at this prompt
are initiated on the logical switch with that FID.
• Use the fosexec --fid FID -cmd "command" to initiate a specific command on a specific logical
switch, where command is the command string.
The fosexec command can be entered for any logical switch to run the specified FOS command on
the specified logical switch, whereas the setcontext command runs only in the current logical switch.

Connecting logical switches

A logical fabric is a fabric that contains at least one logical switch. You can connect logical switches to
non-virtual fabrics switches and to other logical switches using two methods:
• Through ISLs. For extension traffic, the ISL connection is through a tunnel.
• Through base switches and extended ISLs (XISLs). This is supported by the Brocade FX8-24 blade
and Brocade 7840 switch. Refer to Enabling XISL for VE_Ports (FX8-24 blade / 7840 switch) on
page 102.

For more information on virtual fabrics

For more detail on managing and configuring virtual fabrics, refer to chapter on managing Virtual
Fabrics in the Fabric OS Administrator's Guide.

Considerations for logical switches

Before creating IPIFs, IP routes, tunnels, and circuits, follow procedures for creating logical switches
as outlined in Logical switch overview on page 94 and as detailed in the chapter on managing virtual
fabrics in the Fabric OS Administrator's Guide. Use the following information and instructions for
creating tunnels and other components on logical switches.

Ethernet port sharing

In Fabric OS v 7.0 and later, VE_Ports in different logical switches can share a single Ethernet port (1
GbE, 10 GbE, or 40 GbE) located on the default switch. As a best practice, leave Ethernet interfaces
in the default switch even if you will only use a single virtual fabric logical switch. If new VF logical
switches are added and need to use the Ethernet interface, then the Ethernet interface doesn't have to
be moved back to the default switch.

NOTE
For Fabric OS versions prior to Fabric OS v7.0, in order to use a Ethernet port for a tunnel, that port
must be in the same logical switch as the tunnel's VE_Port.
With Ethernet port sharing, you can have the following configuration, as an example:

96 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Limitations of Ethernet port sharing

• Default switch has port GbE0

• Logical switch 1 has VE17, which has a circuit over GbE0
• Logical switch 2 has VE18, which also has a circuit over GbE0
All of the committed-rate restrictions and bandwidth sharing of the Ethernet ports for ARL remain the
same for shared ports in the logical switches. VE_Ports created from shared Ethernet ports initiate as
regular VE ISLs in their respective logical switches.
When IPIFs are created for physical ports (including crossports) located in the default switch, these IP
interfaces can be used by circuits assigned to tunnels created in other logical switches. This means that
multiple VE_Ports in multiple logical switches can use the same Ethernet port. Although multiple circuits
can use the same Ethernet port, these circuits can be differentiated in the IP network using VLAN tags
or access control lists (ACLs) set for the source and destination IP addresses in the circuit. Refer to
Managing QoS, DSCP, and VLANs on page 103 for more information on using VLAN tagging for
Extension features.

Limitations of Ethernet port sharing

Note the following limitations of port sharing:

• Only Ethernet ports in the default switch can be shared by VE_Ports in different logical switches. A
Ethernet port in a non-default switch can only be used by VE_Ports in that same logical switch.
• The GbE ports in other logical switches or ports on the base switch cannot be shared by ports in
different logical switches.
• Tunnels created on 7800 switches and FX8-24 blades with a mix of dedicated ports (ports within the
same logical switch) and shared ports (ports in the default switch) are not supported.
• When using shared Ethernet interfaces between the default switch and other logical switches, if the
default switch is disabled, the Ethernet ports in the default switch will also be disabled. This will
impact all tunnels in the other logical switches using the Ethernet interfaces.

Port sharing example

This section illustrates an example of port sharing on an FX8-24 blade. The following output for the
portshow ipif all command illustrates IP interfaces, IP routes, and crossports configured for ports in the
default logical switch and tunnels and circuits on two different logical switches that use these
configurations.
Note the following about the configuration detailed in the output:
• This example is for port sharing configuration on a FX8-24 blade.
• There are three logical switches:
‐ LS 0 has FID 128 and is the default switch.
‐ LS 2 has FID 50.
‐ LS 4 has FID 70.
• IP interfaces and IP routes for these IPIFs were created for xge0 and xge1. The portcfg - -ipif and
portcfg - - iproute commands were issued in the default logical switch context where the ports
reside. Refer to Configuring IPIFs and IP routes on page 99 for more information.
• Crossports were configured for both xge0 and xge1 on the default switch. Refer to Crossports on
page 41 for more information.
• A tunnel with VE_Port 22 and circuits was created on LS 2. VE_Port 22 was first moved to LS 2, and
the portcfg fciptunnel commands to configure the tunnel and circuits were issued in the context for
LS 2 (FID 50). Refer to Moving ports between logical switches on page 100 and Configuring tunnels
and circuits on page 99 for more information.
• A tunnel with VE_Port 12 and circuits was created on LS 4. VE_Port 12 was first moved to LS 4, and
the portcfg fciptunnel commands to configure the tunnel and circuits were issued in the context for

Brocade Fabric OS Extension Administrator's Guide 97

53-1003507-05
Configuring Extension Features

LS 4 (FID 70). Refer to Moving ports between logical switches on page 100 and Configuring
tunnels and circuits on page 99 for more information.
CURRENT CONTEXT -- LS: 0, FID: 128 *NOTE this
is the default switch.*
switch:admin> portshow ipif

Port IP Address / Pfx MTU VLAN Flags

--------------------------------------------------------------------------------
8/xge0 10.108.0.90 / 24 1500 n/a U R M
8/xge0 10.108.0.91 / 24 1500 n/a U R M
8/xge0 10.108.0.92 / 24 1500 n/a U R M X
8/xge0 10.108.0.93 / 24 1500 n/a U R M X
8/xge1 10.108.1.90 / 24 1320 n/a U R M
8/xge1 10.108.1.91 / 24 1320 n/a U R M
8/xge1 10.108.1.92 / 24 1320 n/a U R M X
8/xge1 10.108.1.93 / 24 1320 n/a U R M X
--------------------------------------------------------------------------------
Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running
N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp X=Crossport

switch:admin> portshow iproute

Port IP Address / Pfx Gateway Flags

--------------------------------------------------------------------------------
8/xge0 10.108.0.0 / 24 * U C
8/xge0 10.108.0.91 / 32 * U C
8/xge0 10.108.100.0 / 24 10.108.0.250 U G S
8/xge0 10.108.0.0 / 24 * U C X
8/xge0 10.108.0.93 / 32 * U C X
8/xge0 10.108.100.0 / 24 10.108.0.250 U G S X
8/xge1 10.108.1.0 / 24 * U C
8/xge1 10.108.1.91 / 32 * U C
8/xge1 10.108.101.0 / 24 10.108.1.250 U G S
8/xge1 10.108.1.0 / 24 * U C X
8/xge1 10.108.1.93 / 32 * U C X
8/xge1 10.108.101.0 / 24 10.108.1.250 U G S X
--------------------------------------------------------------------------------
Flags: U=Usable G=Gateway H=Host C=Created(Interface)
S=Static L=LinkLayer X=Crossport

CURRENT CONTEXT -- LS: 2, FID: 50 *Note that this is one of the logical
switches (not the default switch).*
portshow fciptunnel all -c:
-------------------------------------------------------------------------------
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met
-------------------------------------------------------------------------------
1/22 - Up cft---- 14d18h 226.60 2.73 5 - -
1/22 0 1/xge0 Up ---4v-s 7d17h34m 64.80 0.78 7 1000/3000 0
1/22 1 1/xge0 Up ---4v-s 7d5h24m 48.59 0.59 7 1000/2000 0
1/22 2 1/xge1 Up ---4vxs 7d17h34m 64.60 0.78 7 1000/3000 0
1/22 3 1/xge1 Up ---4vxs 7d5h24m 48.60 0.58 7 1000/2000 0
-------------------------------------------------------------------------------
Flags (tunnel): M=MainTunnel L=LocalBackup R=RemoteBackup
i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
A=AdvCompr L=LZCompr d=DeflateCompr D=AggrDeflateCompr
(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU 4=IPv4 6=IPv6
ARL a=Auto r=Reset s=StepDown t=TimedStepDown

CURRENT CONTEXT -- LS: 4, FID: 70 *Note that this is a different logical switch
(and not the default switch).*
portshow fciptunnel all -c :
-------------------------------------------------------------------------------
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met
-------------------------------------------------------------------------------
1/12 - Up c--F--- 19d15h 0.00 0.00 1 - -
1/12 0 1/xge0 Up ---4vxs 7d17h34m 0.00 0.00 3 1000/3000 0
1/12 1 1/xge0 Up ---4vxs 7d5h24m 0.00 0.00 4 1000/2000 1
1/12 2 1/xge1 Up ---4v-s 7d17h34m 0.00 0.00 3 1000/3000 0
1/12 3 1/xge1 Up ---4v-s 7d5h24m 0.00 0.00 4 1000/2000 1
-------------------------------------------------------------------------------
Flags: tunnel: c=compression m=moderate compression a=aggressive compression
A=Auto compression f=fastwrite t=Tapepipelining F=FICON
T=TPerf i=IPSec l=IPSec Legacy
Flags: circuit: s=sack v=VLAN Tagged x=crossport 4=IPv4 6=IPv6
L=Listener I=Initiator

98 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring IP interfaces and IP routes

Configuring IP interfaces and IP routes

The following example configures IP interfaces (ipif) and IP routes (iproutes) for ports that reside on the
default switch and creating tunnels and circuits on a different logical switch that use these IP interfaces.
You must issue the portcfg ipif and portcfg iproute commands in the logical switch context where the
Ethernet port resides. If the Ethernet port is in the default switch, then the commands must be entered
from the default switch context. If the Ethernet ports are in a logical switch other than the default switch,
you must issue the commands in that context. In the latter case, the Ethernet ports cannot be used by
tunnels created in any other logical switch in the chassis.
In the following example, port ge0 on an FX8-24 blade in slot 8 of a DCX-4S is on the default switch.
The default switch FID in this case is 128.
1. If you are in a different logical switch context than the default switch, set the context to 128 using the
setcontext 128 command.
sw0:FID60:admin>setcontext 128
2. Enter the portcfg ipif command to create the interface on the port.
sw0:FID128:admin>portcfg ipif 8/ge0 create 192.168.1.24 netmask 255.255.255.0 mtu
1500
3. Configure an IP route if necessary using the portcfg iproute command in the FID 128 context.
The following command creates an IP route to destination network 192.168.11.0 for port ge0 on the
FX8-24 blade in slot 8. The route is through local gateway 192.168.1.1.

sw0:FID128:admin> portcfg iproute 8/ge0 create 192.168.11.0 netmask 255.255.255.0

192.168.1.1

Other than issuing commands for IP interfaces and IP routes from the correct logical switch context,
other aspects of the commands used in this procedure are the same as for any switch. For more
information, refer to Configuring an IPIF on page 79 and Configuring an IP route on page 80.

Configuring tunnels and circuits

To configure a tunnel on a logical switch other than the default switch, you must first move the VE_Port
to the logical switch from the default switch, and then create the tunnel and circuits in that logical switch
context. Issue the portcfg fciptunnel command in the context of the logical switch where the VE_Port
resides. In the following example, the VE_Port resides on the default switch with FID 128. A tunnel has
not been configured yet using this VE_Port.
Following are example steps to configure a tunnel from a Brocade FX8-24 blade to a Brocade 7800
switch. Other than issuing commands to move VE_Ports and to create tunnels and circuits from the
correct logical switch context, other aspects of configuring tunnels and circuits are the same for any
switch. For more information, refer to Creating an Extension tunnel on page 82 and Creating additional
circuits on page 85.
1. Move the VE_Port from the default switch to the logical switch with FID 60. VE_Port 12 is available
through physical port ge0 located on the default switch.
switch:admin> lscfg --config 60 -port 8/12
2. Set the context to the logical switch with FID 60 using the following command.
switch:admin> setcontext 60
3. Create a tunnel endpoint on the new logical switch for the FX8-24 blade using the IP interface
created for port ge0 on the default logical switch for the blade and a destination address for a remote
Brocade 7800 switch. In the following example, the destination address (192.168.11.78) is specified
first, followed by the source address (192.168.1.24). ARL minimum (-b) and maximum (-B) committed
rates are specified for circuit 0, which is the default circuit created automatically when you configure a
tunnel.
switch:FID60:admin> portcfg fciptunnel 8/12 create --remote-ip 192.168.11.78 --
local-ip 192.168.1.24 -b 15500 -B 62000

Brocade Fabric OS Extension Administrator's Guide 99

53-1003507-05
Moving ports between logical switches

4. Create a tunnel endpoint on the Brocade 7800 switch using the portcfg fciptunnel command. Note
that the Brocade 7800 switch is not enabled for Virtual Fabrics.
switch:admin> portcfg fciptunnel 16 create --remote-ip 192.168.1.24 --local-ip
192.168.11.78 -b 15500 -B 62000
5. Create an additional circuit for the FX8-24 end of the tunnel using the following command.
switch:FID60:admin> portcfg fcipcircuit 8/12 create 1 --remote-ip 192.168.11.78 --
local-ip 192.168.1.25 -b 15500 -B 62000
6. Create the circuit on the Brocade 7800 end of the tunnel using the following command.
switch:admin> portcfg fcipcircuit 16 create 1 --remote-ip 192.168.1.25 --local-ip
192.168.11.78 -b 15500 -B 62000

Moving ports between logical switches

To move ports between logical switches, use the following command:
lscfg --config FID -port slot/port
• The FID variable is the Fabric ID of the logical switch where port is moving to.
• The slot number is required for the Brocade FX8-24 blade. Omitted on the Brocade 7800 and the
Brocade 7840 switch.
• The port number is the FC, VE, or GE port number. For the Brocade 7800 switch, GbE ports are
ge0 through ge5. For the Brocade FX8-24 blade, XGE (10 GbE) ports are xge0 or xge1 and GbE
ports are ge0-ge9. For the Brocade 7840 switch, 40 GbE ports are ge0-1 and 10 GbE ports are ge2
through ge17.
The following are considerations for moving ports between logical switches:
• The 1 GbE ports (Brocade 7800 switch and FX8-24 blade), 10 GbE ports (FX8-24 blade and
Brocade 7840 switch), 40 GbE ports (Brocade 7840 switch only), and VE_Ports can be part of any
logical switch. They can be moved between any two logical switches unless they are members of a
circuit configuration.
• Because Ethernet ports and VE_Ports are independent of each other, both must be moved in
independent steps. You must delete the configuration on VE_Ports and Ethernet ports before
moving them between logical switches.
• You must move a VE_Port from the logical switch where it resides to a new logical switch in order to
create a tunnel for the new logical switch.

Displaying logical switch configurations

You can display the logical switch configuration for a switch and the Ethernet ports located in each
logical switch using the lscfg --show -ge command. The following output shows that all Ethernet ports
are located in the default switch (FID 128).

DCX68:FID128:root> lscfg --show -ge

Created switches: 128(ds) 10 60 68 127
Slot 1 2 3 4 5 6 7 8 9 10 11 12
--------------------------------------------------------------------------
Port
0 | 128 | | | | | | | | | | | 128 |
1 | 128 | | | | | | | | | | | 128 |
2 | 128 | | | | | | | | | | | 128 |
3 | 128 | | | | | | | | | | | 128 |
4 | 128 | | | | | | | | | | | 128 |
5 | 128 | | | | | | | | | | | 128 |
6 | 128 | | | | | | | | | | | 128 |
7 | 128 | | | | | | | | | | | 128 |
8 | 128 | | | | | | | | | | | 128 |
9 | 128 | | | | | | | | | | | 128 |
10 | 128 | | | | | | | | | | | 128 |
11 | 128 | | | | | | | | | | | 128 |

100 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade 7800 switch considerations and limitations

You can display the logical switch configuration and the VE_Ports assigned to each logical switch using
the lscfg --show command. The following output shows that besides the default switch with FID 128,
other default switches have been created with FID 10, 60, 68, and 127.
Note that some of the VE_Ports for the FX8-24 blade in slot 1 have been moved from the default switch
to other logical switches.

DCX68:FID128:root> lscfg --show

Created switches: 128(ds) 10 60 68 127
Slot 1 2 3 4 5 6 7 8 9 10 11 12
--------------------------------------------------------------------------
Port
0 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
1 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
2 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
3 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
4 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
5 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
6 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
7 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
8 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
9 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
10 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
11 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
12 | 60 | | | | 128 | | | 128 | 60 | | | 60 |
13 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
14 | 10 | | | | 128 | | | 128 | 60 | | | 128 |
15 | 128 | | | | 128 | | | 128 | 60 | | | 128 |
16 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
17 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
18 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
19 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
20 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
21 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
22 | 10 | | | | 128 | | | 128 | 10 | | | 128 |
23 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
24 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
25 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
26 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
27 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
28 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
29 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
30 | 128 | | | | 128 | | | 128 | 10 | | | 128 |
31 | 128 | | | | 128 | | | 128 | 10 | | | 128 |

Brocade 7800 switch considerations and limitations

The following are considerations and limitations for Brocade 7800 switches configured to support Virtual
Fabrics:
• Although you can create up to four logical switches on a Brocade 7800 switch, a base switch cannot
be created. Therefore, you cannot use the logical switches for XISLs.
• Up to four logical switches will support FICON CUP; however, refer to your system qualification letter-
specific limits.
• FCR is not supported on a Brocade 7800 switch enabled with logical switches because the Brocade
7800 has no base switch to support EX_Ports.
• A Brocade 7800 switch configured with multiple logical switches cannot be downgraded to a release
prior to FOS 7.1.0 without deleting all of the non-default logical switches and configurations.

Brocade FX8-24 blade considerations and limitations

The following are considerations and limitations of FX8-24 blades configured to support Virtual Fabrics:
• The number of logical switches that you can create and the limits on logical switch support for FICON
CUP depends on the chassis where the FX8-24 is installed. For example, up to eight logical switches

Brocade Fabric OS Extension Administrator's Guide 101

53-1003507-05
Enabling XISL for VE_Ports (FX8-24 blade / 7840 switch)

can be configured on a blade installed on DCX 8510 platforms. Refer to your chassis specifications
for details.
• For FX8-24 blade, you can make the logical switch a base switch if you are planning on using an
extended interswitch link (XISL) connection between base switches instead of using separate ISL
connections from logical switches.

Enabling XISL for VE_Ports (FX8-24 blade / 7840 switch)

Another way to connect logical switches is to use extended interswitch links (XISLs) and base
switches. When you divide a chassis or fixed-port switch into logical switches, you can designate one
of the switches to be a base switch. A base switch is a special logical switch that is used for
interconnecting the physical chassis.
An XISL connection can be created between base switches, instead of using separate ISLs. The base
fabric provides the physical connectivity across which logical connectivity will be established. The XISL
can carry combined traffic for multiple logical fabrics while maintaining traffic separation for each
fabric.
Because of the expense of long-distance links, this feature has particular benefit for the Brocade
extension platforms. This feature is supported on tunnels between Brocade FX8-24 blades running
Fabric OS v7.0 and later, and on tunnels between 7840 switches running Fabric OS v7.4.0.
To create a base switch on the Brocade FX8-24 blade, use the -base option for the lsCfg command
when creating a logical switch. To use XISL, refer to instructions for configuring a logical switch to use
XISLs in the Fabric OS Administrator's Guide.
For the Brocade FX8-24 blade, if an XISL is enabled, it is recommended that you do not configure
VE_Ports on both the logical switch and the base switch because tunnels support only two hops
maximum.

Brocade 7840 switch considerations and limitations

Following are considerations and limitations of Brocade 7840 switches configured to support Virtual
Fabrics:
• For Brocade 7840 switch, you can make the logical switch a base switch if you are planning on
using an extended interswitch link (XISL) connection between base switches instead of using
separate ISL connections from logical switches.
• Up to four logical switches will support FICON CUP; however, refer to your system qualification
letter-specific limits.
• A tunnel from a Brocade 7840 switch requires one of the following:
‐ A VE_Port and Ethernet port in the same logical switch.
‐ A VE_Port in a logical switch and shared Ethernet port in the default switch (best practice).

NOTE
When in 10VE mode, all the unused VE_Ports must be in the default switch, VE_Ports 29-33 and
39-43. If the unused VE_Ports are not in the default switch, the VE-Mode cannot be set to 10VE
mode. Unused VE_Ports cannot be moved to other logical switches while in 10VE mode.

102 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Managing QoS, DSCP, and VLANs

Managing QoS, DSCP, and VLANs

Quality of Service (QoS) refers to policies for handling differences in data traffic. These policies are
based on data characteristics and delivery requirements. For example, ordinary data traffic is tolerant of
delays and dropped packets, but real-time voice and video data are not. QoS policies provide a
framework for accommodating these differences in data as it passes through a network.
QoS for Fibre Channel traffic is provided through internal QoS priorities. Those priorities can be mapped
to TCP/IP network priorities using zone name prefixes and VCs. The different priority TCP sessions can
be marked upon egress. The TCP marking is done at the IP layer using Layer 3 Differentiated Services
Code Point (DSCP) or at the Ethernet layer within the 802.1Q tag header using 802.1P. There are two
options for TCP/IP network-based QoS:
• DSCP
• VLAN tagging and Layer 2 Class of Service (L2CoS)
You can configure QoS, DSCP, and VLAN tagging at the tunnel and circuit level for data path traffic.

DSCP Quality of Service

Layer 3 Class of Service Differentiated Services Code Point (DSCP) refers to a specific implementation
for establishing QoS policies as defined by RFC 2475. DSCP uses six bits of the Type of Service (TOS)
field in the IP header to establish up to 64 different values to associate with data traffic priority.
DSCP settings are useful only if IP routers are configured to enforce QoS policies uniformly within the
network. IP routers use the DSCP value as an index into a Per Hop Behavior (PHB) table. Control
connections and data connections can be configured with different DSCP values. Before configuring
DSCP settings, determine if the IP network you are using implements PHB, and consult with the WAN
administrator to determine the appropriate DSCP values.

VLANs and Layer 2 Quality of Service

Devices in physical LANs are constrained by LAN boundaries. They are usually in close proximity to
each other, and share the same broadcast and multicast domains. Physical LANs often contain devices
and applications that have no logical relationship. Also, when logically related devices and applications
reside in separate LAN domains, they must be routed from one domain to the other.
A virtual local area network (VLAN) can reside within a single physical network, or it can span several
physical networks. Related devices and applications that are separated by physical LAN boundaries
can reside in the same VLAN. Also, a large physical network can be broken into smaller VLANs. VLAN
traffic is routed using 802.1Q-compliant tags within an Ethernet frame. The tag includes a unique VLAN
ID, and Class of Service (CoS) priority bits. The CoS priority scheme (also called Layer 2 Class of
Service or L2CoS) uses three Class of Service (CoS or 802.1P) priority bits, allowing eight priorities.
Consult with your WAN administrator to determine usage.

Managing DSCP and VLAN support on circuits

When VLAN tag is created on a circuit, all traffic over that circuit will use the specified VLAN. VLAN
tagging of ingress traffic is often used to identify multiple circuits of different VE_Ports coming from a 10
GbE port into a 10 GbE port on a switch or router. If the switch or router has its 10 GbE interface
configured as a VLAN trunk and each circuit has its own tagging, then the switch or router can direct the
traffic accordingly. Only traffic flow over the configured circuit will be tagged with the VLAN tagging and
the L2CoS and DSCP options. The L2CoS and DSCP markings can be configured with command
options such as --l2cos-f-class, --l2cos-high, --dscp-f-class, --dscp-high.

Brocade Fabric OS Extension Administrator's Guide 103

53-1003507-05
VLAN tagging examples

For details on the options available on the portcfg fciptunnel command to enable VLAN support on
circuit 0 and on the portcfg fcipcircuit command for additional circuits, refer to Fabric OS Command
Reference.

VLAN tagging examples

The following example shows the VLAN tag option on the portcfg fciptunnel create command. The
VLAN tag applies only to circuit 0 because a circuit was not identified and this is the default circuit for a
tunnel.

NOTE
The -v vlanid option is available for the Brocade 7800 switch and the FX8-24 blade. For the Brocade
7840 switch, the VLAN ID is set for the IPIF using the portcfg ipif command.

switch:admin> portcfg fciptunnel 16 create --remote-ip 192.168.2.20 --local-ip

192.168.2.10 -b 100000 -B 150000 -v 100
Operation Succeeded

The following example creates an additional circuit with a different VLAN tag.

switch:admin> portcfg fcipcircuit 16 create 1 --remote-ip 192.168.2.21 --local-ip

192.168.2.11 -b 100000 -B 150000 -v 200
Operation Succeeded

The following example shows the portcfg fcipcircuit modify command that changes the VLAN tag
and L2CoS levels for circuit 0. Parameters are the same for both the create and modify options.

switch:admin> portcfg fcipcircuit 16 modify 0 -v 300 --l2cos-f-class 7 --l2cos-high

5 --l2cos-medium 3 --l2cos-low 1

The following example shows the portcfg fcipcircuit modify command that changes the DSCP
values for circuit 0. Parameters are the same for both the create and modify options.

switch:admin> portcfg fcipcircuit 16 modify 0 --dscp-f 32 --dscp-h 16 --dscp-m 8 --

dscp-l 4
Operation Succeeded

The following example shows the use of the portshow command to display the tunnel and circuit
values. Use the -c option as shown to include circuit values.

switch:admin> portshow fciptunnel 16 -c

When both DSCP and L2CoS are used

If a tunnel or circuit is VLAN tagged, both DSCP and L2CoS may be tagged on ingress traffic unless
the VLAN is end-to-end with no intermediate hops in the IP network. The following table shows DSCP
priorities mapped to L2CoS priorities. This may be helpful when consulting with the network
administrator. You can modify DSCP and L2CoS values for different priority traffic when configuring
circuits for extension switches and blades.

104 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Managing the VLAN tag table

TABLE 15 Default mapping of DSCP priorities to L2CoS priorities

DSCP priority/bits L2CoS priority/bits Assigned to

7 / 000111 1 / 001 Medium QoS

11 / 001011 3 / 011 Medium QoS

15 / 001111 3 / 011 Medium QoS

19 / 010011 3 / 011 Medium QoS

23 / 010111 3 / 011 Medium QoS

27 / 011011 0 / 000 Class 3 Multicast

31 / 011111 0 / 000 Broadcast/Multicast

35 / 100011 0 / 000 Low Qos

39 / 100111 0 / 000 Low Qos

43 / 101011 4 / 100 High QoS

46 / 101110 7 / 111 Class F

47 / 101111 4 / 100 High QoS

51 / 110011 4 / 100 High QoS

55 / 110111 4 / 100 High QoS

59 / 111011 4 / 100 High QoS

63 / 111111 0 / 000 Reserved

Managing the VLAN tag table

The VLAN tag table is used by ingress processing to filter inbound VLAN tagged frames per IP
interface. The table is used to determine how to tag a frame that is not already tagged. If a VLAN
tagged frame is received from the network and there is no entry in the VLAN tag table for the VLAN ID,
the frame is discarded. The per-IP interface VLAN configuration is for non-data path traffic only, such as
ICMP or ping commands. If Class F traffic or data path traffic needs to be tagged, it must be done using
the following methods:

Brocade Fabric OS Extension Administrator's Guide 105

53-1003507-05
Configuring Extension Features

• For the Brocade 7800 switch and FX8-24 blade, use the -v, - -vlan-tagging options in the portcfg
fcipcircuit create or portcfg fcipcircuit modify commands.
• For the 7840 switch, set the VLAN tag value using the vlan vlan_id option in the portcfg ipif
command.
To tag frames destined for a specific host address, you must create an entry with an exact matching
destination address in the table. Only frames destined for that address are tagged with the associated
VLAN ID. To tag frames destined for a specific network, you must create a destination address entry
for the network. For example, if a destination address of 192.168.100.0 is specified, then all frames
destined for the 192.168.100.0 network are tagged with the associated VLAN ID, assuming a network
mask of 255.255.255.0. If frames are already VLAN tagged, those tags take precedence over entries
in this table.

NOTE
If you do not specify a destination IP address, the destination address defaults to 0.0.0.0, and all
frames are tagged with the associated VLAN tag.
1. Connect to the switch and log in using an account assigned to the admin role.
2. For the Brocade 7800 and FX8-24 blade, enter the portCfg vlantag command to add or delete
entries in the VLAN tag table. The general syntax for the command is as follows:
portCfg vlantag [add|delete] ipif_addr vlan_id L2CoS dst_IP_addr
For full details on syntax and using this command, refer to the Fabric OS Command Reference.
The following example adds an entry that tags all frames from IP address 192.168.10.1 destined for
IP address 192.168.20.1 with a VLAN ID of 100, and a L2CoS value of 3.

switch:admin> portcfg vlantag 8/ge0 add 192.168.10.1 100 3 192.168.20.1

The following example for the FX8-24 blade adds an entry that tags all frames from a crossport with
local address 192.168.11.20, VLAN ID of 200, and a LSCoS value of 1.

switch:admin> portcfg vlantag 8/xge0 add 192.168.11.20 200 1 -x

3. For the Brocade 7840 switch, add entries in the VLAN tag table using the vlan vlan-id option in the
portcfg ipif command, which is used to configure an IP interface (IPIF) for circuits that you intend
to configure on a Ethernet port. If no VLAN ID is specified for the IP address, no ID is used. The IP
address used on the command line will belong to the VLAN ID specified.
The general syntax for using this option in the command is as follows:
portcfg ipif slot/port .dp0|1 create src_ipaddr vlan vlan_id
The following example adds an entry that tags all frames from IP address 192.168.1.10 with a
VLAN ID of 100.
portcfg ipif ge1.dp0 create 192.168.1.10/24 vlan 100

Note that because GbE ports are shared between DP0 and DP1 on the Brocade 7840 switch, the
ge1.dp0 specifies the DP where the command should be sent.

4. To display the VLAN tag configuration, use the portCfg vlantag command as follows:
switch:admin> portshow vlantag 8/ge0
switch:admin> portshow vlantag 8/xge0

For more details on using the portCfg vlantag and portshow vlantag commands, refer to the
Fabric OS Command Reference.

106 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Implementing IPsec over tunnels

Implementing IPsec over tunnels

Internet Protocol security (IPsec) uses cryptographic security to ensure private, secure communications
over Internet Protocol networks. IPsec supports network-level data integrity, data confidentiality, data
origin authentication, and replay protection. It helps secure your extension traffic against network-based
attacks from untrusted computers.
The following sequence of events invokes the IPsec protocol.
1. IPsec and Internet Key Exchange (IKE) policies are created and assigned on peer switches or blades
on both ends of the tunnel.
2. IKE negotiates security association (SA) parameters, setting up matching SAs in the peers. Some of
the negotiated SA parameters include encryption and authentication algorithms, Diffie-Hellman key
exchange, and SAs.
3. Data is transferred between IPsec peers based on the IPsec parameters and keys stored in the SA
database.
4. When authentication and IKE negotiation are complete, the IPsec policy is now ready for data traffic.
5. SA lifetimes terminate through deletion or by timing out. An SA lifetime equates to approximately two
billion frames of traffic passed through the SA or after a set time interval has passed.
6. As the SA is about to expire, an IKE re-key will occur to create a new set of SAs on both sides and
data will start using the new SA.

Limitations using IPsec over tunnels

The following limitations apply to using IPsec:
• Network Address Translation (NAT) is not supported.
• Authentication Header (AH) is not supported.
• IPsec-specific statistics are not supported on the Brocade 7800 switch and FX8-24 blade. The
Brocade 7840 does provide statistics.
• IPsec can only be configured on IPv4-based tunnels on the Brocade 7800 switch and FX8-24 blade.
The Brocade 7840 does support IPsec on IPv6-based tunnels.
• Older versions of the FX8-24 blade do not support IPsec on VE_Ports 22-31. For these blades, a
RASlog warning message will display that blade is not at correct version to support IPsec enabled
tunnels on VE_Ports 22 through 31.
• On the Brocade 7800 switch and FX8-24 blade, a tunnel using IPSec on Fabric OS v7.0.0 and later
must use the --legacy tunnel option if connecting to a peer running a version prior to Fabric OS
v7.0.0.
• IPsec is not allowed with the --connection-type tunnel option set to anything other than default.

IPsec for the extension switches and blades

Advanced Encryption Standard, Galois/Counter Mode, Encapsulating Security Payload (AES-GCM-
ESP) is used as a single, predefined mode of operation for protecting all TCP traffic over a tunnel. AES-
GCM-ESP is described in RFC 4106. The following list contains key features of AES-GCM-ESP:
• Encryption is provided by AES with 256-bit keys.
• The IKEv2 key exchange protocol is used by peer switches and blades for mutual authentication.
• IKEv2 uses UDP port 500 to communicate between the peer switches or blades.
• All IKEv2 traffic is protected using AES-GCM-ESP encryption.
• An SHA-512 hash message authentication code (HMAC) is used to check data integrity and detect
third-party tampering.

Brocade Fabric OS Extension Administrator's Guide 107

53-1003507-05
Enabling IPsec and IKE policies

• Pseudo-random function (PRF) is used to strengthen security. The PRF algorithm generates output
that appears to be random data, using the SHA-512 HMAC as the seed value.
• A 2048-bit Diffie-Hellman (DH) group is used for both IKEv2 and IPsec key generation.
• The SA lifetime limits the length of time a key is used. When the SA lifetime expires, a new key is
generated, limiting the amount of time an attacker has to decipher a key. Depending on the length
of time expired or the length of the data being transferred, parts of a message may be protected by
different keys generated as the SA lifetime expires.
For extension switches and blades, the SA lifetime expires after two billion frames of data. The
expiration time will vary, depending on the platform. The lifetime is based upon datagrams that have
been encrypted over the tunnel regardless of the number of bytes or the time that the tunnel has
been up. Once an IPsec SA has been used for 2 billion datagrams, a new SA or re-key sequence is
initiated.
• Encapsulating Security Payload (ESP) is used as the transport mode. ESP uses a hash algorithm to
calculate and verify an authentication value, and only encrypt the IP payload.
• A circuit in a non-secure tunnel can use the same Ethernet interface as a circuit in a secure tunnel.
• Brocade IPsec is a hardware implementation that does not degrade or impact performance.
• Brocade IPsec does not preclude the use of compression or QoS.

Enabling IPsec and IKE policies

IPsec is enabled on the tunnel level, not on the circuit level. For the Brocade 7800 switch and FX8-24
blade, you define and enable IPsec using the --ipsec option of the portcfg fciptunnel create and
portcfg fciptunnel modify commands. The -i option activates IPsec. The -K (preshared-key) option
specifies the IKE key. The -l (legacy) option specifies to use the IPsec connection process compatible
with Fabric OS releases prior to v7.0.0. Note that the -l option is a disruptive modify request that
causes the tunnel to bounce.
On the Brocade 7840 switch, before enabling IPsec on a tunnel, you must first define an IPsec policy
using the portcfg ipsec-policy command. When defining the IPsec policy, use the -K option to specify
the IKE key. Enable the policy on a tunnel using the --ipsec policy option for the portcfg fciptunnel
command. Display the defined IPsec policies, IKE sessions associated with the policy, and other
detailed information using the portshow ipsec-policy command. Display IPsec configuration on a
specific tunnel using the portshow fciptunnel command.
The IKE key must be a shared 32-character string for the Brocade 7800 switch and FX8-24 blade, and
a 12-64 character string for the Brocade 7840 switch. Both ends of the secure tunnel must be
configured with the same key string, referred to as a pre-shared key (PSK). If both ends are not
configured with the same key, the IKE session will not come up, and will prevent the extension tunnel
from coming up.
The following examples are for the Brocade 7800 switch and FX8-24 blade. They show IPsec and IKE
keys enabled for traffic from VE_Ports 16 and 17 across multiple FCIP circuits.

portcfg fciptunnel 16 create --remote-ip 192.168.0.90 --local-ip 192.168.0.80 -b 50000 -B 50000 \

-x 0 -d c0 -I -K12345678901234567890123456789012 -l
portcfg fcipcircuit 16 create 1 --remote-ip 192.168.1.90 --local-ip 192.168.1.80 -b 50000 -B 50000 -x 0
portcfg fcipcircuit 16 create 2 --remote-ip 192.168.2.90 --local-ip 192.168.2.80 -b 50000 -B 50000 -x 0
portcfg fcipcircuit 16 create 3 --remote-ip 192.168.3.90 --local-ip 192.168.3.80 -b 50000 -B 50000 -x 0
portcfg fcipcircuit 16 create 4 --remote-ip 192.168.4.90 --local-ip 192.168.4.80 -b 50000 -B 50000 -x 0
portcfg fcipcircuit 16 create 5 --remote-ip 192.168.5.90 --local-ip 192.168.5.80 -b 50000 -B 50000 -x 0
portcfg fciptunnel 17 create --remote-ip 192.168.0.91 --local-ip 192.168.0.81 -b 50000 -B 50000 -x 0 -
d \
c0 -I -K12345678901234567890123456789012 -l
portcfg fcipcircuit 17 create 1 --remote-ip 192.168.1.91 --local-ip 192.168.1.81 -b 50000 -B 50000 -x 0
portcfg fcipcircuit 17 create 2 --remote-ip 192.168.2.91 --local-ip 192.168.2.81 -b 50000 -B 50000 -x 0
portcfg fcipcircuit 17 create 3 --remote-ip 192.168.3.91 --local-ip 192.168.3.81 -b 50000 -B 50000 -x 0
portcfg fcipcircuit 17 create 4 --remote-ip 192.168.4.91 --local-ip 192.168.4.81 -b 50000 -B 50000 -x 0
portcfg fcipcircuit 17 create 5 --remote-ip 192.168.5.91 --local-ip 192.168.5.81 -b 50000 -B 50000 -x 0

108 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade 7840 IKE authentication failures

The following command creates an IPsec policy for the Brocade 7840 switch.

switch:admin> portcfg ipsec-policy myPolicy1 create -k

“some test key”
Operation Succeeded.

The following command enables the IPsec policy for a Brocade 7840 switch.
switch:admin> portcfg tunnel 24 modify --ipsec myPolicy1
Operation Succeeded.

The following command displays the policy information on the Brocade 7840 tunnel.

switch:admin> portshow fciptunnel -c

Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met
-------------------------------------------------------------------------
24 - Up --i----- 33m4s 0.00 0.00 3 - -
24 0 ge2 Up ---ah--4 33m4s 0.00 0.00 3 1000/1000 0
-------------------------------------------------------------------------
Flags (tunnel): i=IPSec f=Fastwrite T=TapePipelining F=FICON r=ReservedBW
A=AdvCompr L=LZCompr d=DeflateCompr D=AggrDeflateCompr
(circuit): h=HA-Configured v=VLAN-Tagged p=PMTU 4=IPv4 6=IPv6
ARL a=Auto r=Reset s=StepDown t=TimedStepDown

The following example displays IPsec IKE policy on the Brocade 7840 tunnel.

switch:admin> portshow ipsec-policy --ike

IPSec Policy Key

IKE-ID Oper Flg Local-Addr Remote-Addr IKE Rekey ESP Rekey
--------------------------------------------------------------------------------
ipsec1 abcdefghijklmnopqrst
dp0.0 UP I 192.168.0.20 192.168.0.10 5h59m46s 0 3h8m59s 0
dp0.1 UP I 192.168.0.20 192.168.0.11 5h59m49s 0 3h20m28s 0
dp1.0 UP I 192.168.0.21 192.168.0.10 5h59m47s 0 3h2m0s 0
--------------------------------------------------------------------------------
Flags: *=Name Truncated. Use "portshow ipsec-policy -d for details".I=Initiator
R=Responder

Brocade 7840 IKE authentication failures

On the Brocade 7840 switch, an IKE authentication error may require user intervention to correct. This
error occurs if there is an IKE session parameter mismatch. When an IKE authentication error occurs,
the IKE session is put into a faulty state and remains there until manually corrected. This action
prevents an intruder from using multiple attempts to authenticate. For example, if the preshared-key is
not correct during the initial IKE authentication exchange, it will trigger an authentication error. The error
is seen with the RASlog XTUN-2012 message. Refer to Fabric OS Message Reference for details. You
can enter the portshow ipsec-policy --ike command where the IKE Operational Status is reported as
FAULT.
To recover from an IKE authentication error, you must restart the IKE authentication exchange. To
restart the IKE authentication, remove IPSec from the tunnel and add it back in using the portcfg
fciptunnel ve modify --ipsec none and portcfg fciptunnel ve modify --ipsec policy command.
Alternatively you can create another policy with the correct parameters and modify the tunnel to use the
new policy.

NOTE
The policy names do not have to match from end to end, but the policy parameters must match.

Brocade Fabric OS Extension Administrator's Guide 109

53-1003507-05
Traffic Isolation Zoning

Traffic Isolation Zoning

The Traffic Isolation (TI) Zoning feature allows you to control the flow of inter-switch traffic by creating
a dedicated path for traffic flowing from a specific set of source ports (N_Ports). You can use Traffic
Isolation Zoning to ensure that requests and responses for FCIP-based applications such as Open
Systems Tape Pipelining use the same VE_Port tunnel across a metaSAN.
Traffic isolation is implemented using a special zone, called a Traffic Isolation zone (TI zone). A TI
zone indicates the set of N_Ports, E_Ports, and VE_Ports to be used for a specific traffic flow. When a
TI zone is activated, the fabric attempts to isolate all inter-switch traffic entering from a member of the
zone to only those E_Ports that have been included in the zone. The fabric also attempts to exclude
traffic not in the TI zone from using E_Ports or VE_Ports within that TI zone.
For more information and details to configure TI Zoning, refer to the "Traffic Isolation Zoning" chapter
in the Fabric OS Administrator's Guide.

110 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
Configuring IP Extension features

● Configuration preparation for IP Extension features......................................................111

● Configuration steps for IP Extension features............................................................... 112
● Configuring hybrid mode for IP Extension features....................................................... 113
● Configuring GbE port for IP Extension LAN features.................................................... 114
● Configuring LAG............................................................................................................ 115
● Configuring switch virtual interface IPIF........................................................................ 116
● Configuring a tunnel to support IP Extension................................................................ 117
● Configuring bandwidth distribution................................................................................ 118
● Configuring tunnel compression.................................................................................... 119
● Configuring traffic control lists....................................................................................... 120

Configuration preparation for IP Extension features

IP Extension features are available when you configure the Brocade 7840 switch to operate in hybrid
mode. When operating in hybrid mode, you can configure most tunnel features associated with tunnels,
and additionally configure IP Extension features. Most tunnel or circuit configurations done with the
switch not in hybrid mode are carried over to hybrid mode.
Before you begin to configure IP Extension features, do the following:
• Follow the recommendations in Configuration preparation on page 73 to prepare the WAN support.
WAN support must be completed prior to configuring IP Extension features on the Brocade 7840
switch.
• To use the IP Extension features, the Brocade 7840 switch must operate in hybrid mode.
• For the Brocade 7840 switch, you must configure the VE_Port operating mode for 10VE mode.
• FIPS is not supported when the Brocade 7840 operates in hybrid mode.

NOTE
Once enabled, FIPS cannot be disabled.
• Determine which Ethernet ports will be used for LAN connectivity. Ensure these ports are in the
default switch and configured to LAN mode. The Ethernet ports on the Brocade 7840 switch are in
groups and connections should be spread across the groups and not within the groups if possible.
• For the Brocade 7840 device, the IP MTU size must be at least 1280. If the supported maximum IP
MTU size in the network is larger than 9216, the IP MTU of the Brocade 7840 should be 9216.
• Identify the subnets that will be extended through the Brocade 7840 switch. Assign IP addresses,
subnet masks, and MTUs to be used as LAN gateways to the 7840.
• When the Brocade 7840 switch is operating in hybrid mode and IP Extension features are enabled,
you must configure traffic control lists (TCLs).
• Determine the VE_Port numbers you want to use. The VE_Port numbers serve as tunnel IDs.
Typically, the first one is used.
• Determine how many additional circuits you want to create. You will need the source and destination
IP addresses for each circuit, and the minimum and maximum rates for ARL, or the committed rate if
not using ARL. You will need to know if you intend to assign metrics to circuits so that lower metric
circuits fail over to circuits with higher metrics. For all circuits except circuit 0, these values are set by
the portCfg fcipcircuit create command.

Brocade Fabric OS Extension Administrator's Guide 111

53-1003507-05
Configuration steps for IP Extension features

• When configuring tunnels to support large numbers of devices, consider memory limitations of the
extension switch or blade if you are enabling any type of emulation feature, such as FCP or FICON.
If too many devices are present or activated at one time, acceleration operations can be negatively
impacted. Refer to Memory use limitations for large-device tunnel configurations on page 27.
• You must consider the maximum limit of LAN connections allowed, which is 512 TCP connections
and 64 non-TCP (UDP) connections.
• When planning for VE_Port and bandwidth usage, consider which VE_Port is hosted by which DP
complex. This affects load balancing between each DP complex.

Configuration steps for IP Extension features

Use the following major steps for configuring IP Extension features on extension switches.
1. Configure the operating mode on the Brocade 7840 switch for 10VE. In hybrid mode, the default is
10VE operating mode. 20VE mode is not allowed in hybrid mode.
2. Configure the Brocade 7840 switch to operate in hybrid mode using the extncfg --app-mode
hybrid command.

NOTE
Configuring the switch for hybrid mode is disruptive. The switch reboots and loads the hybrid image.
3. Persistently disable VE_Ports.
4. Create an IP interface (IPIF) for each circuit that you want on a port by assigning an IP address,
netmask, and an IP MTU size to an Ethernet port using the portCfg ipif command. Refer to
Configuring an IPIF on page 79. Note that this step applies to overall WAN configuration and is not
specific to IP Extension LAN configuration on the Brocade 7840 switch.
5. Create one or more IP routes to a port if required using the portCfg iproute command. Refer to
Configuring an IP route on page 80. Note that this step applies to overall WAN configuration and is
not specific to IP Extension LAN configuration on the Brocade 7840 switch.
6. Test the IP connection using the portCmd --ping command. Note that this step applies to overall
WAN configuration and is not specific to IP Extension LAN configuration on the Brocade 7840
switch.

NOTE
If a VLAN is present in the Brocade 7840 switch, it only needs to be configured on the IPIF. In
addition, the VLANs need only match with the local Ethernet hop configured. Refer to Managing the
VLAN tag table on page 105 for details.

NOTE
Stacked VLAN tagging is not supported on the Brocade 7840 switch.
7. Create extension tunnels using the portCfg fciptunnel command. Refer to Creating an Extension
tunnel on page 82. Note that this step applies to overall WAN configuration and is not specific to IP
Extension configuration on the Brocade 7840 switch.
8. Create FCIP circuits (after circuit 0) and enable or disable features using the portCfg fcipcircuit
command. Refer to Creating additional circuits on page 85. Note that this step applies to overall
WAN configuration and is not specific to IP Extension configuration on the Brocade 7840 switch.
9. Identify the subnets that will be extended through the Brocade 7840 switch. Assign IP addresses,
subnet masks, and MTUs to be used as LAN gateways to the 7840.

112 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring hybrid mode for IP Extension features

NOTE
The local-side LANs that contain the end devices must be on different subnets. For example, you
cannot have end-devices on DC-A subnet 10.0.0.0/24 communicating with end-devices DC-B subnet
10.0.0.0/24. Those are the same subnets on each side.
The following steps are specific to configuring IP Extension features after the switch is running in hybrid
mode.
1. Configure a GE port for LAN mode using the portCfgGe command.
2. Configure the GE port for link aggregation group (LAG) operation using the portcfg lag command.
(This step is optional.)
3. Configure a switch virtual interface (SVI) to provide IP access for VLAN using the portcfg ipif
command.
4. Configure a tunnel to support IP Extension using the portcfg fciptunnel command.
5. Configure bandwidth distribution using the portcfg fciptunnel command. The bandwidth distribution
determines how QoS traffic is distributed between FC and IP bandwidth.
6. Configure compression mode for IP Extension using the portcfg fciptunnel command. Note that IP
traffic compression cannot be set to fast-deflate.
7. Configure the traffic control list (TCL) for the port using the portcfg tcl command. A TCL consists of
a rule name, a priority, an input filter, and a target for the rule.
8. Persistently enable the VE_Ports.

Configuring hybrid mode for IP Extension features

Configure the Brocade 7840 switch to operate in hybrid mode, which supports FCIP tunnel features and
IP Extension features.
The Brocade 7840 cannot be running FIPS.
The Brocade 7840 must be in 10VE mode before you can enable hybrid mode. Changing VE modes is
disruptive as it requires rebooting the switch.

NOTE
Configuring the switch for hybrid mode is disruptive. The switch reboots and loads the hybrid image.

NOTE
If you configured tunnels and circuits while in FCIP mode, that configuration is carried over to hybrid
mode if the configuration is compatible with hybrid mode. If the configuration is not valid for hybrid mode
and 10VE mode, you will be prompted to make changes.

The following steps are required to configure the Brocade 7840 to operate in hybrid mode.
1. Use the extncfg --app-mode command to enable hybrid mode. When prompted, confirm the
command action.
switch:admin> extncfg --app-mode hybrid

This action will configure the system for Hybrid (FCIP/IPExt) mode.

WARNING: This is a disruptive operation that requires a reboot to take effect.

Would you like to continue (Y,y,N,n): [ n] y

Brocade Fabric OS Extension Administrator's Guide 113

53-1003507-05
Configuring GbE port for IP Extension LAN features

Operation succeeded. Rebooting the system...

2. Use the extncfg --show command to confirm hybrid mode is enabled.
switch:admin> extncfg --show
APP Mode is Hybrid (FCIP with IPEXT)
VE-Mode: configured for 10VE mode.
switch:admin>

The next task is to configure a GE port for LAN mode.

Configuring GbE port for IP Extension LAN features

Configure the GbE port for IP Extension LAN features. The GbE port must be in LAN mode before it
can use IP Extension features.
The Brocade 7840 must be in hybrid mode.
To use the IP Extension features, you must configure a GbE port to operate in LAN mode. Once a port
is configured as a LAN port, it cannot be used as a WAN port for any circuit definitions.
Any existing IP configuration must be removed before changing a GbE port to LAN mode.
Only 1/10 GbE ports can be configured as LAN ports. 40GbE ports cannot be configured as LAN
ports. The following steps are required to configure a GbE port to operate in LAN mode.
1. Use the portcfgge command to configure a GbE port for LAN operation.
This example puts port GE10 in LAN mode.

switch:admin> portcfgge ge10 --set -lan

Operation Succeeded.
2. Use the portcfgge --show command to verify the GbE port is in LAN mode.
(The example output is truncated.)

switch:admin> portcfgge --show

Port Speed Flags LAG-ID

-----------------------------------------
ge0 40G A---- -
ge1 40G A---- -
ge2 10G A---- -
ge3 10G A---- -
ge4 10G A---- -
ge5 10G A---- -
ge6 10G A---- -
ge7 1G ----- -
ge8 10G A---- -
ge9 10G A---- -
ge10 10G A-L-- -
[ . . . ]
------------------------------------------
Flags: A:Auto-Negotiation Enabled C:Copper Media Type
L:LAN Port G=LAG Member
3. Alternatively, use the switchshow command to verify the GbE port is in LAN mode.
(The example output is truncated.)

switch:admin> switchshow
Index Port Address Media Speed State Proto
==================================================
ge0 id 40G Online FCIP
ge1 id 40G Online FCIP
ge2 id 10G Online FCIP
ge3 id 10G Online FCIP

114 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring LAG

ge4 id 10G Online FCIP

ge5 id 10G Online FCIP
ge6 id 10G Online LAN
ge7 id 10G Online LAN
ge8 id 10G Online LAN
ge9 id 10G Online FCIP
ge10 id 1G Online FCIP

The next optional step is to create one or more link access groups (LAGs) and add ports to the LAG.

Configuring LAG
Configure LAG on a GbE LAN port. A link aggregation group (LAG) treats multiple connections between
two components logically as a single connection. This task is optional.
The GbE port must be configured to operate in LAN mode.
When you create a LAG, you assign a name to it. Ports can be added to and removed from a named
LAG. The port speed and auto-negotiate parameters must match for all ports being added to a LAG.
Port speed and link auto-negotiation can be set for the LAG to control the setting of the individual LAG
member ports. The port speed and link auto-negotiation setting of LAG member ports cannot be set
individually. You can enable or disable individual ports in a LAG.
The following steps configure a static link aggregation group (LAG) for a GbE LAN port.
1. Use the portcfg lag name --create command to create a LAG.
switch:admin> portcfg lag lag0 --create
Operation Succeeded
2. Use the portcfg lag name --add command to add ports to the LAG. You can add ports singly or in a
range.
switch:admin> portcfg lag lag0 --add ge2-ge4,ge7
WARNING: While making configuration changes the modified LAN GE ports will be
disabled. Please run "portenable" command to manually enable the modified LAN GE
ports after completing all the configuration changes.

Would you like to continue (Y,y,N,n): [ n] y

Operation Succeeded
3. Use the portcfg lag name --remove command to remove ports from a LAG.
switch:admin> portcfg lag lag0 --remove ge7
WARNING: While making configuration changes the modified LAN GE ports will be
disabled. Please run "portenable" command to manually enable the modified LAN GE
ports after completing all the configuration changes.

Would you like to continue (Y,y,N,n): [ n] y

Operation Succeeded
4. Use the portcfg lag name --set command to set a speed for ports in the LAG.
switch:admin> portcfg lag edgeSw1 --set -speed 10G
WARNING: While making configuration changes the modified LAN GE ports will be
disabled. Please run "portenable" command to manually enable the modified LAN GE
ports after completing all the configuration changes.

Would you like to continue (Y,y,N,n): [ n] y

Operation Succeeded
5. Use the portcfg lag name --enable command to enable auto-negotiation.
switch:admin> portcfg lag edgeSw1 --enable -autoneg
WARNING: While making configuration changes the modified LAN GE ports will be
disabled. Please run "portenable" command to manually enable the modified LAN GE
ports after completing all the configuration changes.

Would you like to continue (Y,y,N,n): [ n] y

Operation Succeeded

Brocade Fabric OS Extension Administrator's Guide 115

53-1003507-05
Configuring switch virtual interface IPIF

6. Use the portdisable command to disable ports in a LAG.

switch:admin> portdisable ge7
7. Use the portenable command to enable ports in a LAG. You must enable ports after any
modification with the portcfg lag command.
switch:admin> portenable ge7
8. Use the portcfgge --show command to display GbE port status and to see which LAG, if any,
contains a GbE port.
(The example output is truncated.)

switch:admin> portcfgge --show

Port Speed Flags LAG-ID

-----------------------------------------
ge0 40G A---- -
ge1 40G A---- -
ge2 10G A-LG- lag0
ge3 10G A-LG- lag0
ge4 10G A-LG- lag0
ge5 10G A---- -
ge6 10G A---- -
[ . . . ]
------------------------------------------
Flags: A:Auto-Negotiation Enabled C:Copper Media Type
L:LAN Port G=LAG Member
9. Use the portshow command to display LAG status. You can display general LAG status or detailed
LAG status.
a) Use the portshow lag command to display general LAG status.
switch:admin> portshow lag

LAG Name State Port Count Member Ports

---------------------------------------------------------------
lag0 Online 4 GE2,GE3,GE4,GE7
---------------------------------------------------------------

b) Use the portshow lag --detail command to display detailed LAG status.
switch:admin> portshow lag --detail

LAG: lag0
------------------------------------------
Oper State: Online
Port Count: 3
Port AdminSt OperState Speed AutoNeg
ge2 Enabled Online 10G Enabled
ge3 Enabled Online 10G Enabled
ge4 Enabled Online 10G Enabled

The next recommended step is to configure the switch virtual interface (SVI) IP interface.

Configuring switch virtual interface IPIF

Configure the switch virtual interface (SVI) IP interface (IPIF) for the LAN ports.
The Brocade 7840 must be configured for hybrid mode. GE ports must be configured for LAN
operation. Optionally, LAGs are defined.
There is only one SVI interface per data processor (DP) complex, meaning one LAN-side Ethernet
device and MAC per DP. There can be multiple SVI IP addresses defined per DP, but they all use the
same single SVI interface.

116 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring a tunnel to support IP Extension

By using a SVI per DP, the SVI represents all LAN-side ports. This function provides a single routing
table for all LAN-side traffic as well as a single MAC interface per DP for all LAN-side traffic. You can
configure multiple SVI IP addresses on this interface in the same way you provide multiple IP interface
definitions per front-end port on the WAN side. The SVI IP interface behaves much the same way as do
non-SVI IPIF configurations.
The SVI address must be configured on a DP that has VE_Ports configured, which extends the LAN
traffic. If VE_Port is configured on DP0 for a particular LAN traffic, SVI must be configured on that same
DP.
You can configure a maximum of eight SVI IP addresses per DP. You cannot configure more than one
SVI IP address per subnet on a DP.
This SVI IP interface acts as the gateway address for the device(s) being redirected through the
Brocade 7840.
This task is required.
1. Use the portcfg ipif to configure a SVI LAN port on a DP.
The example shows two SVI ports configured and the maximum MTU configured.

switch:admin> portcfg ipif lan.dp0 create 10.0.0.1/24 vlan 100

switch:admin> portcfg ipif lan.dp0 create 10.0.1.1/24 vlan 200 mtu 9216
2. Use the portshow ipif to display the IPIF ports.
The example shows two SVI LAN ports configured on DP0, as well as other GE ports configured on
DP0.

switch:admin> portshow ipif

Port IP Address / Pfx MTU VLAN Flags

------------------------------------------------------------------------------
ge4.dp0 192.168.60.20 / 24 1500 0 U R M
ge17.dp0 192.168.10.107 / 24 1500 0 U R M
lan.dp0 10.0.0.1 / 24 1500 100 U R M
lan.dp0 10.0.1.1 / 24 9216 200 U R M
------------------------------------------------------------------------------
Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running
N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp X=Crossport

The next recommended task is to configure a tunnel to support IP Extension features.

Configuring a tunnel to support IP Extension

Configure a tunnel to support IP Extension features.
The Brocade 7840 must be in hybrid mode. GE ports must be configured for LAN operation. Optionally,
LAGs are defined.
To use IP Extension features, you enable IP Extension capability on a tunnel. You can create an IP
Extension capable tunnel or you can modify an existing tunnel to support IP Extension.
This task is required.
1. Use the portcfg fciptunnel ve-port create command to create an IP Extension capable tunnel.
This example creates and enables an IP Extension tunnel on VE_Port 24.

switch:admin> portcfg fciptunnel 24 create --ipext enable

Brocade Fabric OS Extension Administrator's Guide 117

53-1003507-05
Configuring bandwidth distribution

2. Use the portcfg fciptunnel ve-port modify command to modify an existing tunnel to be IP
Extension capable.
This example modifies an existing tunnel on VE_Port 24 to disable IP Extension.

switch:admin> portcfg fciptunnel 24 modify --ipext disable

The optional next task is to configure bandwidth distribution.

Configuring bandwidth distribution

Tunnel traffic is distributed between a Fibre Channel traffic group and an IP Extension traffic group.
You can configure bandwidth ratios between the two groups when a tunnel is configured to support IP
Extension. QoS priority is distributed within the bandwidth allocations.
The Brocade 7840 must be in hybrid mode. GE ports must be configured for LAN operation.
Optionally, LAGs are defined. A tunnel must be configured to support IP Extension.
When you configure bandwidth distribution, you can change the default allocation of 50/50 between
Fibre Channel (FC) and IP Extension (IP) traffic in a tunnel. All bandwidth allocations are expressed as
percentages.

NOTE
The minimum percentage allowed for a QoS priority or a distribution group cannot go below 10%.
After configuring the FC and IP bandwidth distribution you can configure QoS high, medium, and low
priorities in each of the FC and IP distributions. The default priority values for QoS are 50/30/20 for
high, medium, and low.

NOTE
Creating or modifying distribution bandwidth can disrupt traffic on the specified tunnel for a brief period
of time. The tunnel is brought down before the new configuration is applied, then the tunnel is brought
up.

This task is optional.

1. Use the portcfg fciptunnel create --distribution protocol command to create protocol bandwidth
distribution.
This command creates protocol bandwidth distribution at 60% for FC and 40% for IP traffic.

switch:admin> portcfg fciptunnel 24 create --distribution protocol,60,40

2. Use the portcfg fciptunnel modify --distribution command to change traffic protocol distribution.
The first value applies to FC traffic and the second value applies to IP traffic.
This command modifies protocol bandwidth values to 40% for FC traffic and 60% for IP traffic.

switch:admin> portcfg fciptunnel 24 modify --distribution protocol,40,60

3. Use the portcfg fciptunnel modify command to change the QoS priority bandwidth allocations in
the FC traffic group and the IP traffic group.

118 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring tunnel compression

a) Use the portcfg fciptunnel modify --fc-qos-ratio command to configure the FC QoS priorities.
This command modifies the FC QoS priority bandwidth values for high to 30%, medium to 50%,
and low to 20%.

switch:admin> portcfg fciptunnel 24 modify --fc-qos-ratio 30,50,20

b) Use the portcfg fciptunnel modify --ip-qos-ratio command to configure the IP QoS priorities.
This command modifies IP QoS priority bandwidth values for high to 60%, medium to 30%, and
low to 10%.

switch:admin> portcfg fciptunnel 24 modify --ip-qos-ratio 60,30,10

The optional next task is to configure tunnel compression.

Configuring tunnel compression

Configure tunnel compression per protocol on the tunnel. You can override inherited compression
values.
The Brocade 7840 must be in hybrid mode. GE ports must be configured for LAN operation. A tunnel
must be configured to support IP Extension.
The enhancements for IP Extension allow you to configure compression on the tunnel at a protocol
level. The protocol compression options override the main tunnel compression level and set the
compression for the specified protocol to the desired mode. The available modes depend on the
protocol, whether FC or IP.
The IP priorities do not support fast-deflate compression, so only the deflate and aggressive deflate
options are allowed with the --ip-compression option. If the main tunnel compression is set to fast-
deflate, the IP priorities are set to none. The protocol-based compression modes can be set to default,
which causes the protocol compression to inherit the configuration from the main tunnel compression
setting.
The configuration steps show how to set compression for a tunnel and how to set protocol overrides for
traffic in that tunnel.
This task is optional.
1. Use the portcfg fciptunnel modify --compression command to configure fast-deflate compression
level for a tunnel compression.
a) Configure the fast-deflate tunnel compression.
switch:admin> portcfg fciptunnel 24 modify --compression fast-deflate

b) Verify fast-deflate compression is not supported for IP protocol.

switch:admin> portshow fciptunnel 24

Tunnel: VE-Port:24 (idx:0)

====================================================
...
Compression : Fast Deflate
FC-Compression : Fast Deflate (Inherited)
IP-Compression : None (Inherited)

2. Use the portcfg fciptunnel modify --ip-compression command to change the IP compression to
deflate.

Brocade Fabric OS Extension Administrator's Guide 119

53-1003507-05
Configuring traffic control lists

a) Configure deflate compression for the IP protocol.

switch:admin> portcfg fciptunnel 24 modify --ip-compression deflate

b) Verify deflate compression is configured for IP protocol.

switch:admin> portshow fciptunnel 24

Tunnel: VE-Port:24 (idx:0)

====================================================
...
Compression : Fast Deflate
FC-Compression : Fast Deflate (Inherited)
IP-Compression : Deflate (Override)
...

3. You can use the portcfg fciptunnel modify command to return compression values to their default,
inherited values.
switch:admin> portcfg fciptunnel 24 modify --fc-compr default --ip-compr default

The next task is to configure traffic control lists.

Configuring traffic control lists

Configure traffic control lists in order to manage IP Extension LAN flows.
The Brocade 7840 must be in hybrid mode. GE ports must be configured for LAN operation.
Optionally, LAGs are defined. Only IP Extension tunnels can be associated with a TCL.
To help manage the IP Extension LAN flows, Traffic Control List (TCL) rules are configured by the
user. Each TCL rule is identified by a user-provided name. The name is used as a reference to the
TCL rule and can be used in other CLIs as needed. Rules are created, modified, and deleted by name.
Each TCL name must be a unique value in the switch.
When defining a TCL rule, the TCL action and TCL target will determine the behavior with regard to
the DPs.
For the TCL action, following actions are possible:
• When the action is set to "allow," the target must be an IP Extension-enabled VE tunnel.
• When the action is set to "deny," the rule is pushed to both DPs to deny the matching traffic.
• You can select a specific DP for the rule, DP0 or DP1. In this instance, the rule is pushed to the
specified DP to deny the matching traffic on that DP only.
TCL rules consist of three main parts: rule priority, input filter, and target.
The TCL priority provides an order of precedence to the TCL rule within the overall TCL list. The
priority value must be a unique integer value. You can modify the priority to reposition the TCL rule in a
different location in the list. When removing a rule, or even when creating a new rule, you can leave
gaps in the priority numbers.

NOTE
The priority value must be unique across all active TCL rules. If a TCL is defined as priority 10, that
same priority cannot be used for another TCL rule, even if that rule would be assigned to the other DP.
A check is performed when the rule is enabled to ensure the priority value is unique.

120 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Configuring IP Extension features

The TCL input filter identifies a particular host, device, or application. The input filter is a set of
parameters to help identify the input traffic. It is based on several of the fields found in the IP / TCP and
other protocol headers. Use the portcfg tcl --help command to display the options.
The TCL target parameters determine how to handle the input traffic. Typically the parameters of the
target will identify a specific VE tunnel and a priority. For example, --target 24-high. If no priority is
specified, the input traffic will be sent over the IP medium priority QoS for the target tunnel.
This task is required. You must configure traffic control lists.
A default TCL is always created when the Brocade 7840 switch is configured for hybrid mode. It has
priority 65535, which is the lowest possible, and is set to deny all traffic. This rule cannot be deleted or
modified. If no other rule is created, all traffic will be dropped. Therefore, it is critical that you create at
least one TCL rule and set it to target an IP Extension enabled VE tunnel.
The configuration steps show how to create a TCL named HostAtoB, enable it, identify a target, source
address, and set the rule priority all with a single command.
1. Use the portcfg tcl command to create a TCL.
switch:admin> portcfg tcl hostAtoB create --admin enable --target 24 --src-addr
10.0.0.0/8 --priority 10
Operation Succeeded

2. Use the portshow tcl command to display TCL information.

switch:admin> portshow tcl

Pri Name Flgs Target L2COS VLAN DSCP Proto Port Hit
Src-Addr Dst-Addr
------------------------------------------------------------------------------
*10 hostAtoB AI--- 24-Med ANY ANY ANY ANY ANY 0
10.0.0.0/8 ANY

*65535 default D---- - ANY ANY ANY ANY ANY 0

ANY ANY

------------------------------------------------------------------------------
Flags: *=Enabled ..=Name Truncated (see --detail for full name)
A=Allow D=Deny I=IP-Ext P=Segment Preservation

Brocade Fabric OS Extension Administrator's Guide 121

53-1003507-05
Configuring traffic control lists

122 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
Management and Troubleshooting

● In-band management.................................................................................................... 123

● WAN performance analysis tools.................................................................................. 129
● Using the portshow command.......................................................................................137
● Tunnel issues ............................................................................................................... 142
● Troubleshooting Extension links....................................................................................143
● Using FTRACE..............................................................................................................144

In-band management

NOTE
In-band management is supported on the Brocade 7800 switch and FX8-24 blade only.
In-band management allows management of an extension switch or blade in conjunction with FCIP
traffic through Ethernet ports. This enables a management station located on the WAN side of the
Brocade 7800 switch or FX8-24 blade platform to communicate with the control processor (CP) for
management tasks, such as SNMP polling, SNMP traps, troubleshooting, and configuration. Through IP
forwarding, inband management also allows a management station connected to the management port
of one extension switch or blade to manage the switch or blade at the far end of the network through the
WAN.
The in-band management path is achieved by receiving the management traffic from the Ethernet port
and transmitting the traffic to the CP through the inband interface. The CP then handles the
management traffic as it would handle any other management requests from a normal management
interface. The in-band management interface is protocol-independent, so any traffic destined for these
in-band management interfaces passes through the data processor (DP) to the CP. It is then handled on
the CP according to the rules set forth for the normal management interface and follows any security
rules that may be in place on the CP.
One in-band management interface can be configured per Ethernet interface to provide redundancy.
This allows the management station on the WAN side of the network to have multiple addresses for
reaching that switch and provides redundancy if one of the Ethernet ports cannot be reached.
Communication is handled through external addresses configured independently for each in-band
management interface.
The following functions are not supported by the in-band management interface:
• Downloading firmware
• IPv6 addressing

IP routing
The in-band management interfaces are separate from the existing IP interfaces currently used for
extension tunnel traffic. These interfaces exist on the CP and are added and maintained on the CP
routing table to ensure end-to-end connectivity. Because this routing table will be shared among all
devices on the CP, including the management interface, precautions must be taken to ensure that
proper connectivity is maintained. To ensure proper handling of routes, the in-band management

Brocade Fabric OS Extension Administrator's Guide 123

53-1003507-05
Configuring IP addresses and routes

devices should be configured on a different network from the management interface and from every
other in-band management interface.
In-band management interface addresses must also be unique and cannot be duplicates of any
addresses defined on the Ethernet ports. An in-band management address can exist on the same
network as an address defined on one of the GbE ports because the in-band management interfaces
use the CP routing table and not the routing table normally used for the GbE ports.

Configuring IP addresses and routes

Configure and view IP addresses and routes for in-band management interfaces by using the following
Fabric OS commands:
• portcfg mgmtif slot/gePort create|delete ipAddress netmask mtu
• portcfg mgmtif slot/gePort enable|disable
• portshow mgmtif slot/gePort
• portcfg mgmtroute slot/gePort destination netmask gateway

Management station on the same subnet example

The following figure illustrates an example of configuring in-band management with the management
station attached to the same subnet as managed switches. Note that only the IP address is required
for each extension switch.

FIGURE 21 Management station configured on the same subnet

7800 LI
Configure the in-band management interfaces.

portcfg mgmtif ge0 create 192.168.3.10 255.255.255.0

7800 RI

124 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Management station on a different subnet example

Configure the in-band management interfaces.

portcfg mgmtif ge0 create 192.168.3.20 255.255.255.0

Management Workstation

telnet 192.168.3.10

Management station on a different subnet example

The following figure illustrates an example configuration consisting of switches and the management
station on different networks and attached through a WAN cloud. The routers are assumed to already
have route entries to access each other's subnet.

FIGURE 22 Management station configured on different subnets

7800 L1
• Configure the in-band management interfaces.
portcfg mgmtif ge0 create 192.168.1.10 255.255.255.0
• Configure the in-band management route for the management station.
portcfg mgmtroute ge0 create 192.168.3.0 255.255.255.0 192.168.1.250

Brocade Fabric OS Extension Administrator's Guide 125

53-1003507-05
Redundant connections to the management stations example

7800 R1
• Configure the in-band management interfaces.
portcfg mgmtif ge0 create 192.168.2.20 255.255.255.0
• Configure the in-band management route for the management station.
portcfg mgmtroute ge0 create 192.168.3.0 255.255.255.0 192.168.2.250

Management station
• Add route entries to access the Brocade 7800 external in-band management interfaces.
route add 192.168.1.0 netmask 255.255.255.0 gw 192.168.3.250
route add 192.168.2.0 netmask 255.255.255.0 gw 192.168.3.250
• Access the Brocade 7800 switches through the external in-band management interfaces.
telnet 192.168.1.10

Redundant connections to the management stations example

In the following figure, because the in-band management interfaces do not support a multi-homing
stack, unique addresses must be used on the management station to communicate with different in-
band management interfaces. If both management station interfaces are on the same subnet, then
host-specific routes must be added on the Brocade 7800 switches.

126 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Management and Troubleshooting

FIGURE 23 Redundant connections to management station

7800 L1
• Configure the in-band management interfaces.
portcfg mgmtif ge0 create 192.168.1.10 255.255.255.0
portcfg mgmtif ge1 create 192.168.4.10 255.255.255.0
• Configure the in-band management route for the management workstation.
portcfg mgmtif ge0 create 192.168.1.10 255.255.255.255 192.168.1.250
portcfg mgmtif ge1 create 192.168.4.10 255.255.255.255 192.168.4.250

7800 R1
• Configure the in-band management interfaces.
portcfg mgmtif ge0 create 192.168.2.20 255.255.255.0
portcfg mgmtif ge1 create 192.168.5.20 255.255.255.0
• Configure the in-band management route for the management workstation.
portcfg mgmtroute ge0 create 192.168.3.30 255.255.255.255 192.168.2.250
portcfg mgmtroute ge1 create 192.168.3.31 255.255.255.255 192.168.5.250

Management Workstation

Brocade Fabric OS Extension Administrator's Guide 127

53-1003507-05
VLAN tagging support

• Add route entries to get to the Brocade 7800 external in-band management interfaces.
route add 192.168.1.0 netmask 255.255.255.0 gw 192.168.3.250
route add 192.168.2.0 netmask 255.255.255.0 gw 192.168.3.250
route add 192.168.4.0 netmask 255.255.255.0 gw 192.168.3.250
route add 192.168.5.0 netmask 255.255.255.0 gw 192.168.3.250
• Access the Brocade 7800 switches through the external in-band management interfaces.
telnet 192.168.1.10

VLAN tagging support

To add VLAN tag entries to the VLAN tag table for in-band management interfaces, use the --mgmt or
-m option with the portcfg vlantag command. Complete the following steps.
1. Configure an IP address and route for in-band management interface using the following command
format.
portcfg mgmtif [slot/]ge_port [create|delete] ipAddress netmask mtu
2. Add the VLAN tag entry for the management interface using the following command format.
portcfg vlantag [slot/]ge_port [add|delete] ipAddress L2COS--mgmt

IP forwarding support
IP forwarding is supported over in-band management to allow communication to the remote switch
through the WAN connection. This is done by enabling IP forwarding to allow IP packets arriving at the
CP interface to be forwarded through the in-band management interface to the remote side. To
prevent network routing and actual bridging of the LAN side of the network to the WAN side of the
network, the forwarding rules of the ipfilter command will default to deny any forwarding traffic. To
allow forwarding, new ipfilter command rules must be added to specific destinations. This will prevent
any unintended network traffic from being forwarded from the LAN side to the WAN side of the
network.
The following figure shows an example network where the management station is located on the LAN
side of 7800 L1. Using in-band management, the station can also communicate with 7800 R1.

FIGURE 24 In-band management with IPv4 forwarding

For this example, you must configure the following:

128 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Using the ipfilter command

• On the management station:

‐ IP address 10.1.1.1/24 (defined)
‐ IP route to 192.168.3.20/32 via 10.1.1.10
• On the 7800 L1:
‐ CP management address 10.1.1.10/24
‐ In-band management address 192.168.3.10/24
‐ IP filter forward rule with destination IP address 192.168.3.20
• On the 7800 R1:
‐ CP management address 10.1.2.20/24
‐ In-band management address 192.168.3.20/24
‐ In-band management route to 10.1.1.1/32 via 192.168.3.10
Once all of these configurations are complete, proper IP connectivity should occur through the network.
In the case where there are routed networks between the Brocade 7800 switches, you will need to add
in-band management routes to each Brocade 7800 switch. Using host-specific routes will help eliminate
undesired traffic. If network routes are needed, they can be substituted, but you should note that this will
allow anything on that network to be forwarded, which could result in undesired disruption of traffic.

NOTE
In all routed network cases, all intermediate hops must have route entries to get to the endpoints.

Using the ipfilter command

Use the ipfilter command to create and manage forwarding rules for use with in-band management.
For full details on this command, options, and arguments, refer to the "ipfilter" section of the Fabric OS
Command Reference.
To create an IP forwarding rule, you must first create a new policy if one has not yet been created. The
easiest way to do this is with the --clone option to create a copy of the default policy.

ipfilter --clone inband_ipv4 -from default_ipv4

A new rule can be added to allow forwarding traffic.

ipfilter --addrule inband_ipv4 -rule rule_number

-dp dest_port
-proto protocol
-act [permit|deny] -type FWD -dip destination_IP

Valid dest_port values are any TCP or UDP port numbers or a range of port numbers that you want
forwarded. Valid protocol values are tcp or udp . The destination_IP is the IP address of the in-band
management interface on the remote side. After a rule is added, save the policy and activate it using the
--save and --activate options of the ipfilter command. There can only be a single IPv4 policy active at
any time. Each policy can consist of multiple rules.

WAN performance analysis tools

WAN analysis tools are designed to test connections, trace routes, and estimate the end-to-end IP path
performance characteristics between a pair of Brocade extension port endpoints. These tools are
available as options on the portCmd command. The following options are available:

Brocade Fabric OS Extension Administrator's Guide 129

53-1003507-05
The tperf option

• portCmd --Tperf. A tunnel test tool that generates and sends test data over a tunnel to determine
the characteristics and reliability of the IP network used by the tunnel at the circuit level. Supported
on the Brocade 7800 and FX8-24.
• portCmd --ping. Tests connections between a local Ethernet port and a destination IP address.
• portCmd --traceroute. Traces routes from a local Ethernet port to a destination IP address.
• portCmd --wtool. Generates traffic over a pair of IP addresses to test the link for issues such as
maximum throughput, congestion, loss percentage, out of order delivery, and other network
conditions. Supported on the Brocade 7840.
• portShow fcipTunnel --perf. Displays performance statistics generated from the WAN analysis.

The tperf option

Tperf (portCmd --tperf) is a utility that generates data between a local and remote switch over a
tunnel. It reports the data generated and response from the remote switch to determine characteristics
and reliability of the IP network used by the tunnel.
Tperf operates with a pair of Brocade 7800 switches or FX8-24 blades. One switch or blade plays the
role of a data sink and the other switch or blade plays the role of the data source. During the data
generation process, traffic flows from the source to the sink, then the sink responds to this traffic. The
process continues for a duration that you specify with command options or until you terminate (Ctrl +
C).
Normally, you should establish one Telnet or SSH session for the tperf source and one for the tperf
sink. Also, open additional Telnet or SSH sessions so that you can periodically display TCP connection
statistics using the -tcp or -p options of the portshow fciptunnel slot/veport command. These
statistics can sometimes help you understand the tunnel bandwidth and IP network infrastructure
capability.
To use Tperf, you must first create a tunnel with at least one circuit or modify an existing tunnel using
the Tperf flag -T. As with any tunnel, this must be done on both switches. The following commands
create a Tperf-enabled tunnel with a committed rate of 10000.

portcfg fciptunnel 16 create --remote-ip 192.168.10.1 --local-ip 192.168.10.2 10000 -

T
portcfg fciptunnel 16 create --remote-ip 192.168.10.2 --local-ip 192.168.10.1 10000 -
T

Tperf will test single and multiple circuit tunnels. Tperf also tests the different priority connections that
are provided by a tunnel. When a Tperf-enabled tunnel is operative, it is not an active VE_Port.
Fabrics will not merge over an operative Tperf tunnel. To determine if the Tperf tunnel is up, issue the
following command:

switch:admin> portshow fciptunnel -c

-------------------------------------------------------------------------------
Tunnel Circuit OpStatus Flags Uptime TxMBps RxMBps ConnCnt CommRt Met
-------------------------------------------------------------------------------
16 - Up ----T-- 1h21m43s 0.00 0.00 2 - -
16 0 ge0 Up ---4--s 1h21m34s 0.00 0.00 2 500/500 0
16 1 ge1 Up ---4--s 1h21m43s 0.00 0.00 2 500/500 0
-------------------------------------------------------------------------------
Flags: tunnel: c=compression m=moderate compression a=aggressive compression
A=Auto compression f=fastwrite t=Tapepipelining F=FICON
T=TPerf i=IPSec l=IPSec Legacy
Flags: circuit: s=sack v=VLAN Tagged x=crossport 4=IPv4 6=IPv6 T=Test(CPerf)
L=Listener I=Initiator

The previous display shows VE_Port 16 as up, but a switchshow command for that same VE _Port
will show the following:

switch:admin> switchshow | grep 16

16 16 631000 -- -- Offline VE

130 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Using ping to test a connection

For full details on syntax and using this command, refer to the Fabric OS Command Reference.
The following examples create a Tperf data sink and a Tperf data source on VE_Port 16.

switch:admin> portcmd --tperf 16 -sink -interval 15

switch:admin> portcmd --tperf 16 -source -interval 15 -high -medium -low

Tperf generates statistics every 30 seconds by default unless you specify a different value for -interval.

TABLE 16 Tperf output

Item Description

Tunnel ID Numeric identifier for the Tperf tunnel.

Traffic Priority High, Medium, or Low.

bytes tx Number of bytes transmitted.

bytes rx Number of bytes received.

PDUs tx Number of protocol data units transmitted.

PDUs rx Number of protocol data units received.

bad CRC headers rx Number of bad CRC headers received.

bad CRC payloads rx Number of bad CRC payloads received.

out of seq PDUs rx Number of out-of-sequence PDUs received.

flow control count Flow control count.

packet loss (%) The percentage of packet loss.

bytes/second The number of bytes transmitted per second.

last rtt The time it took for the last round-trip between the Tperf source and the Tperf sink in
milliseconds. This is calculated only on the source-side report. It is reported as N/A on the
sink-side report.

Using ping to test a connection

The portCmd --ping command tests the connection between the IP address of a local Ethernet port and
a destination IP address. If you want to use this command to test a VLAN connection when you do not
have an active tunnel, you must manually add entries to the VLAN tag table on both the local and
remote sides of the route, using the portCfg vlantag command.

Brocade Fabric OS Extension Administrator's Guide 131

53-1003507-05
Using traceroute

The general syntax of the portCmd --ping command is as follows:

portCmd --ping slot/ge-port -s source_ip -d destination_ip -n num_request -q diffserv -t -ttl -w
wait_time -z size -v vlan_id -c L2_Cos
On the Brocade 7840 switch, because DP complexes share Ethernet ports, identification for the port is
ge n.DP n, for example ge0.DP0. This directs the command to a specifc DP complex.

Using traceroute
The portCmd traceroute command traces routes from a local Ethernet port to a destination IP
address. If you want to use this command to trace a route across a VLAN when you do not have an
active tunnel, you must manually add entries to the VLAN tag table on both the local and remote sides
of the route using the portCfg vlantag command.
The general syntax of the portCmd --traceroute command is as follows:
portCmd --traceroute slot/ge-port -s source_ip -d destination_ip -h max_hops -f first_ttl -q diffserv -w
wait -time -z size -v vlan_id -c L2_Cos
On the Brocade 7840 switch, since DP complexes share Ethernet ports, identification for the port is ge
n.DP n, for example ge0.DP0. This directs the command to a specifc DP complex.
The following example traces the route between IP addresses 192.168.2.22 and 192.168.2.30 over
VLAN 12 from a 7840 switch.
portcmd --traceroute ge2.dp1 -s 192.168.10.1 -d 192.168.20.1 -v 12

The following example traces the route between IP addresses 192.168.10.1 and 192.168.20.1 over
VLAN 10 from an FX8-24 blade.

portcmd --traceroute 8/ge0 -s 192.168.10.1 -d 192.168.20.1 -v 10

NOTE
To trace a route with crossport addresses, refer to Using traceroute with crossports on page 44.
For details of command syntax and output examples, refer to the Fabric OS Command Reference.

Using WAN Tool

WAN Tool allows you to generate traffic at a specified rate in Kbps over a pair of IP addresses to test
the network link for issues such as maximum throughput, congestion, loss percentage, out of order
delivery, and other network conditions. The main purpose of this tool is to determine the health of a link
before deploying it for use as a circuit in a tunnel.
Following are requirements and considerations for using WAN Tool:
• WAN Tool is supported by the Brocade 7840 switch only.
• A maximum of four WAN Tool sessions are supported per DP complex.
• Each session can support a 10 Gbps connection (maximum).
• A test session can run over an IP path being used by an existing circuit between two switches;
however, you must disable the circuit at each end before configuring the session.
• You must configure the WAN Tool session on the switch at each end of the circuit.
• After configuration, you can start a test from one switch only to test unidirectional traffic to the
opposite switch or you can test bidirectional traffic between both switches using the bidirectional
option. If bidirectional is specified for the test session, you can start the session at either switch.
• You can configure multiple test sessions (one per circuit) for a single port, but the total rate
configured for all sessions must be equal to or less then the physical speed of the port (40 Gbps, 10

132 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 WAN Tool commands

Gbps, or 1 Gbps). For example, on a 10 Gbps port, you could configure four 2.5 Gbps sessions. As
another example, on a 40 Gbps interface, you could configure four 10 Gbps sessions.
• The default MTU size used in the test session is 1500, however jumbo frames are supported.
Increase the MTU size for the IP address pair being tested using the portcfg ipif ge_port create
command. For details on this command, refer to the Fabric OS Command Reference or Configuring
an IPIF on page 79.
A tunnel and WAN Tool cannot operate at the same time since they both utilize the TCP ports 3225 and
3226. Therefore, you must disable the circuit that you are testing at the local and remote switch before
you can configure a WAN Tool connection. When you configure WAN Tool on both switches with the
necessary parameters, non-guaranteed TCP connections are established between the switches.
Issuing the WAN Tool start command starts traffic flow on these connections.
Multiple non-guaranteed TCP connections are established for the WAN Tool session to insure that the
traffic being generated between the IP pair is as balanced as possible. The configured rate is split
equally among 500 Mbps connections. For example, if you configure a 10 Gbps rate for the test
session, twenty 500 Mbps connections are created. As another example, if you configure a 1 Gbps rate,
two 500 Mbps connections are created. If the rate cannot be split equally into 500 Mbps connections,
connections with different rates are created. For example, if you configure a 1.5 Gbps rate, four 375
Mbps connections are created. You can verify these connections are created after configuring WAN
Tool on both switches using the portcmd--wtool wt-id show -c command. Refer to the example output
of this command in Configuring WAN tool and displaying results on page 134.

WAN Tool commands

Configure a WAN Tool session using the portcmd --wtool command. The general syntax for creating a
test session including all command options is as follows:
portcmd --wtool wt-id create --src src_ip --dst dst_ip --rate link_rate --time test_time --bi-directional
--ipsec policy name.
You must configure the following parameters on each switch:
• WAN Tool session test ID (wt-id) - The ID doesn't have to match on each switch, but this is
recommended for easier comparison of test results on both ends of the circuit when multiple test
sessions are created. Valid IDs are 0 through 7.
• Link rate (link_rate) in Kbps - Configure the same link rate on the switch at each end of the circuit.
The WAN Tool connections will not fully establish until the same rate is specified for each switch.
• IPsec policy name (policy name) - The policy name can be different on each switch, the IPsec policy
configuration parameters must be the same on each switch.
• Source IP (src_ip) and destination IP (dst_ip) address - The source address will be the destination
address and the destination address will be the source address on the opposite switch.
• Bi-directional (--bi-directional) - This is an optional parameter, but if used, configure on both
switches.
• Test session time (test_time) - The test duration time in minutes must be configured on at least one
switch, but you do not need to configure the time on both switches nor does it need to match on both
switches. The test session uses the time configured on the switch where the test started. If bi-
directional is specified, the session runs for the time configured on the switch where the test started,
then runs for the time (if configured) configured on the opposite switch.
Modify the link rate, test time, test direction (--bi-directional) parameters, and clear statistics for a WAN
Tool test session after creating a test session, using the portcmd --wtool wt-id modify command.

NOTE
You must stop the WAN Tool session before modifying parameters using portcmd --wtool wt-id stop.
Following are examples of using the modify parameter:

Brocade Fabric OS Extension Administrator's Guide 133

53-1003507-05
Configuring a WAN Tool session and displaying results

• To modify the rate, use portcmd --wtool wt-id modify --rate link_rate.
• To clear test results, use portcmd --wtool wt-id modify --clear.
Start and stop a configured test session on a specific switch using the following commands:
• portcmd --wtool wt-id start. You can specify the test duration using portcmd --wtool wt-id start --
time min if the test duration has not been specified with the create or modify parameters.
• portcmd --wtool wt-id stop
Clear test statistics using the portcmd --wtool wt-id modify --clear command.
Delete test sessions using the portcmd --wtool wt-id delete command. Delete all configured test
sessions using portcmd --wtool all delete. At this point, you can re-enable the circuit for operation in
a tunnel using the portCfg fcipcircuit create command.
Display statistics from a WAN Tool session using the portcmd --wtool wt-id show, where wt-id is the
ID (0-7) you used to create the test session. Display all test sessions (if multiple test sessions are
configured) using the portcmd --wtool all show.
For more details on WAN Tool command and parameters, refer to the Fabric OS Command
Reference.

Configuring a WAN Tool session and displaying results

Use the following steps to configure a WAN tool session and display results.
1. Connect to the switch and log in using an account assigned to the admin role.
2. Disable the circuit for the IP pair that you wish to test at each switch using the portCfg fcipcircuit
modify --admin-status disable command.
The following example disables circuit 1.
Switch1:admin>portCfg fcipcircuit 24 modify 1 --admin-status disable
3. Verify that the circuit is disabled using the portshow fciptunnel -c command. The OpStatus for
circuit 1 should be "Down."
4. Establish a test connection on the circuit by configuring a WAN Tool session on the switch at one
end of the circuit.
The following example configures a test connection (WAN Tool session 0) on circuit 1 between
source IP of 10.1.1.1 and destination IP of 10.1.1.2.
Switch1:admin>portcmd --wtool 0 create --src 10.1.1.1 --dst 10.1.1.2 --rate
10000000
5. Configure the WAN Tool session on the switch at the other end of the circuit.
Switch2:admin>portcmd --wtool 0 create --src 10.1.1.2 --dst 10.1.1.1 --rate
10000000

The wt-id (0) does not need to match configuration on Switch1, but this is recommended for easier
comparison of test results on both ends of the circuit when multiple test sessions are created. The
rate must be the same for both switches. Note that the source address of Switch1 becomes the
destination address for Switch2 and the destination address becomes the source address. Refer to
WAN Tool commands on page 133 for a list of WAN Tool command parameter values that must be
identical for both switches in the circuit.

NOTE
The connection will not complete until a WAN Tool session is configured on both switches with an
identical link rate.

134 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Management and Troubleshooting

6. Verify that the WAN Tool test connection has established using the portcmd --wtool wt-id show
command and portcmd --wtool wt-id show -c command.
Switch1:admin>portcmd --wtool 0 show
wantool-id: (0)
=========================================
State : Established
Up Time : 7m37s
Run Time : 0s
Time remaining : 0s
IP Addr (L/R) : 10.1.1.2 <-> 10.1.1.1
PMTUD : Disabled
Comm Rate : 10000000 Kbps (1220.70 MB/s)
Tx rate : 4562.50 Kbps (0.56 MB/s)
Rx rate : 4539.69 Kbps (0.55 MB/s)
Tx Utilization : 0.05%
Rx Utilization : 0.05%
RTT (Min/Max) : 0.10ms/0.28ms
RTT VAR (Min/Max) : 0.09ms/0.34ms
Local Session Statistics
Tx pkts : 0
Peer Session Statistics
Rx pkts : 0
Ooo pkts : 0
Drop pkts : 0 (0.00%)
Switch1:admin>portcmd --wtool 0 show -c
Id Port(L/R) Rate(Tx/Rx) UpTime RunTime
======================================================================
6 63494 / 3225 0.03 / 0.03 8m8s 0s
17 63490 / 3225 0.03 / 0.03 8m8s 0s
14 63498 / 3225 0.03 / 0.03 8m8s 0s
3 61443 / 3226 0.03 / 0.03 8m8s 0s
11 61447 / 3226 0.03 / 0.03 8m8s 0s
9 61446 / 3226 0.03 / 0.03 8m8s 0s
1 61442 / 3226 0.03 / 0.03 8m8s 0s
20 63491 / 3225 0.03 / 0.03 8m8s 0s
8 63495 / 3225 0.03 / 0.03 8m8s 0s
12 63497 / 3225 0.03 / 0.03 8m8s 0s
4 63493 / 3225 0.03 / 0.03 8m8s 0s
16 63489 / 3225 0.03 / 0.03 8m8s 0s
13 61448 / 3226 0.03 / 0.03 8m8s 0s
19 61440 / 3226 0.03 / 0.03 8m8s 0s
5 61444 / 3226 0.03 / 0.03 8m8s 0s
15 61449 / 3226 0.03 / 0.03 8m8s 0s
7 61445 / 3226 0.03 / 0.03 8m8s 0s
18 61441 / 3226 0.03 / 0.03 8m8s 0s
10 63496 / 3225 0.03 / 0.03 8m8s 0s
2 63492 / 3225 0.03 / 0.03 8m8s 0s
======================================================================
Number of Connections:20

The example output from the --wtool 0 show indicates that the connection has an established state.
The example output from the --wtool 0 show -c command displays the non-guaranteed TCP
connections created between TCP ports 3225 and 3226 to balance the test traffic. For the 10 Gbps
test connection, twenty non-guaranteed TCP connections are created.
7. If you have created multiple WAN Tool sessions, you can verify basic connection information using
the --wtool all show command.
Switch1:admin>portcmd --wtool all show
Id CommRate Port Local IP <-> Remote IP TxMBps RxMBps Drop %
===============================================================================
0 10000000 ge9.dp0 10.1.1.2<->10.1.1.1 0.56 0.55 0.00
4 10000000 ge9.dp1 10.1.2.2<->10.1.2.1 0.56 0.56 0.00
===============================================================================
Output for this example shows that WAN Tool session 0 was created to test the circuit with IP
address pair 10.1.1.2 and 10.1.1.1 and session 4 was created testing the circuit with IP address pair
10.1.2.2 and 10.1.2.1.
8. Start traffic on the test connection by entering the portcmd --wtool wt-id start command. If you did
not specify a test duration with the portcmd --wtool wt-id create command, you can do so with the
start command using the --time option.
Switch1:admin>portcmd --wtool 0 start --time 10

Brocade Fabric OS Extension Administrator's Guide 135

53-1003507-05
Resolving test session problems

9. Verify that the test session started by entering the portcmd --wtool wt-id show command.
Switch1:admin>portcmd --wtool 0 show
State : Running
Up Time : 15m39s
Run Time : 6s
Time remaining : 9m54s
IP Addr (L/R) : 10.1.1.1 <-> 10.1.1.2
PMTUD : Disabled
Comm Rate : 10000000 Kbps (1220.70 MB/s)
Tx rate : 9394147.00 Kbps (1146.75 MB/s)
Rx rate : 6102.34 Kbps (0.74 MB/s)
Tx Utilization : 93.94%
Rx Utilization : 0.06%
RTT (Min/Max) : 2.23ms/3.00ms
RTT VAR (Min/Max) : 0.21ms/1.37ms
Local Session Statistics
Tx pkts : 4766856
Peer Session Statistics
Rx pkts : 0
Ooo pkts : 0
Drop pkts : 0 (0.00%)

Note that the "State" shows that the test is running and other statistics display as well, such as test
"Run Time" and "time remaining".
10.Start the test from the other switch by entering the portcmd --wtool wt-id start --time test_time
command.

NOTE
If you used the bi-directional option when creating the session, you can start the session on either
switch.
Switch2:admin>portcmd --wtool 0 start --time 10
11.Verify that the test session started on the other switch by entering the portcmd --wtool wt-id show
command.
Switch2:admin>portcmd --wtool 0 show
12.Delete the WAN Tool session on both switches using the portcmd --wtool wt-id delete command.
13.To verify that the WAN Tool session is disabled, enter the portcmd --wtool wt-id show command.
14.Enable the circuit from each switch using the portcfg fcipcircuit port create command. The
following example enables the circuit from Switch1.
Switch1:admin>portCfg fcipcircuit 24 modify 1 --admin-status enable

Resolving test session problems

If output from the portcmd --wtool wt-id show command shows that the "State" is down, constantly in
progress, or the connection times out (changes from an up to down state) the WAN Tool test
connection is not being established. Verify that you have configured the session on both switches in
the circuit with appropriate parameters and values. Refer to the list of parameters required for each
switch in WAN Tool commands on page 133.
Common problems in establishing a connection can result from the following WAN Tool configuration
problems:
• The test rate doesn't match on each switch.
• The test rate on a single circuit or multiple circuits on a port is greater than the rate allowed for the
port. Note that this will generate a warning that the bandwidth has been exceeded and blocks you
from creating a session.
• The IPsec policy doesn't match on each switch. Note that the IPsec names do need to match, but
the names must refer to the same policy.
• Configured source and destination IP addresses are not correct on one or both switches.

136 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Using the portshow command

Using the portshow command

Use the portshow command to display port operational information on Brocade extension switches and
blades. The Fabric OS Command Reference provides complete descriptions of the portshow
command syntax and options. The following sections identify a few specific outputs that may be useful
for maintenance and troubleshooting.

Displaying IP interfaces
The following example displays IP interface information for a Brocade 7800 or a Brocade 7840 switch.
switch:admin> portshow ipif ge0

The following example displays IP interface information for an FX8-24 blade.

switch:admin> portshow ipif 1/xge0

Displaying IP routes
The following example displays IP route information for a Brocade 7800 or a Brocade 7840 switch.
switch:admin> portshow iproute ge5

The following example displays IP route information for an FX8-24 blade.

switch:admin> portshow iproute 1/xge0

Displaying switch mode information with the extncfg command

The Brocade 7840 switch operates in hybrid mode to support the IP Extension features. The following
example displays the operating mode for the Brocade 7840.

switch:admin> extncfg --show

Displaying GbE port information with the portcfgge command

The following example displays GbE port configuration for a Brocade 7840.

switch:admin> portcfgge --show

Listing the MAC addresses of LAN and GE ports

Use the portcfgge ge#/lan --show -lmac command to display the MAC addresses of the LAN and GE
ports on the Brocade 7840 Extension Switch. Refer to the following sample outputs for more
information.

Brocade Fabric OS Extension Administrator's Guide 137

53-1003507-05
Displaying LAG information

Examples
The following example lists the MAC addresses of all the LAN and GE ports in the switch:

switch:admin> portcfgge --show -lmac

Port dpid MAC Address

----------------------------------------
ge0 dp0 00:05:33:65:7d:48
ge0 dp1 00:05:33:65:7d:48
ge1 dp0 00:05:33:65:7d:49
ge1 dp1 00:05:33:65:7d:49
ge2 dp0 00:05:33:65:7d:4a
ge2 dp1 00:05:33:65:7d:4a
ge3 dp0 00:05:33:65:7d:4b
ge3 dp1 00:05:33:65:7d:4b
ge4 dp0 00:05:33:65:7d:4c
ge4 dp1 00:05:33:65:7d:4c
ge5 dp0 00:05:33:65:7d:4d
ge5 dp1 00:05:33:65:7d:4d
lan dp0 00:05:33:65:7d:5a
lan dp1 00:05:33:65:7d:6f

The following example lists the MAC addresses of the LAN ports in the switch:

switch:admin> portcfgge lan --show -lmac

Port dpid MAC Address

------------------------------------------
lan dp0 00:05:33:65:7d:5a
lan dp1 00:05:33:65:7d:6f
------------------------------------------

The following example lists the MAC addresses of the GE port numbered #4 in the switch:

switch:admin> portcfgge ge4 --show -lmac

Port dpid MAC Address

------------------------------------------
ge4 dp0 00:05:33:65:7d:4c
ge4 dp1 00:05:33:65:7d:4c
------------------------------------------

Displaying LAG information

You can display link aggregation group (LAG) information for the Brocade 7840 switch only when it is
operating in hybrid mode.
The following example displays LAG information.

switch:admin> portshow lag

You can display more detailed information by using the detail option.

switch:admin> portshow lag --detail

Displaying tunnel information

The following example of the portshow fciptunnel command is used most often to determine tunnel
status.

switch:admin> portshow fciptunnel all -c

138 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Displaying tunnel HCL information

Displaying tunnel HCL information

The following example displays the tunnel HCL status for a Brocade 7840 switch.

switch:admin> portshow fciptunnel --hcl-status

Displaying TCL information

You can display traffic control list (TCL) configuration information for the Brocade 7840 switch only when
it is operating in hybrid mode.
The following example displays the TCL configuration information.

switch:admin> portshow tcl

You can display more detailed information by using the detail option.

switch:admin> portshow tcl --detail

Displaying IP Extension LAN statistics

You can display IP Extension LAN statistics for a Brocade 7840 switch only when it is operating in
hybrid mode.
The following example displays the global LAN statistics for a Brocade 7840.

switch:admin> portshow lan-stats --global

The following example displays the active per-flow LAN statistics for a Brocade 7840 switch.

switch:admin> portshow lan-stats --per-flow

The following example displays the most recently closed per-flow LAN statistics for a Brocade 7840
switch.

switch:admin> portshow lan-stats --hist-stats

The following example displays the known applications for a Brocade 7840 switch.

switch:admin> portshow lan-stats --known-apps

App Port-Id(s)
--------------------------------------------------------------------------------
CIFS 139,445
FCIP 3225-3226
FTP 20-21,989-990,115
HTTP 80,8080,8000-8001,3128
HTTPS 443
iSCSI 3260
Isilon-SyncIQ 5666-5667
LDAP 389,8404,636
MS-SQL 1443
MySQL 3306
NETAPP-SNAP-MIRROR 10566
NFS 2049
ORACLE-SQL 66,1525,1521
RSYNC 873

Brocade Fabric OS Extension Administrator's Guide 139

53-1003507-05
Displaying performance statistics

SRDF 1748
SSH 22
SSL-SHELL 614
TELNET 23,107,513,992
TFTP 69
VERITAS-BACKUP 6101-6102,6106,3527,1125
VTS-GRID Control 1415-1416
VTS-GRID Data 350
--------------------------------------------------------------------------------

Displaying performance statistics

Display a summary of performance statistics for tunnels and circuits using the circuit, perf, and
summary options as in the following example.

switch:admin> portshow fciptunnel all --circuit --perf --summary

Displaying QoS statistics

Display QoS statistics for tunnels using the --qos and --summary and options as in the following
example.

switch:admin> portshow fciptunnel all --qos --summary

Displaying details
You can display configuration details using the detail option with the all option as in the following
example.

switch:admin> portshow fciptunnel all --detail

Displaying tunnel information (Brocade 7800)

The following example will display general tunnel information related to port 16 on a Brocade 7800
switch.
switch:admin> portshow fciptunnel 16

Displaying a tunnel with circuit information

The following example adds circuit information to the portshow fciptunnel command output using the
-c option.
switch:admin> portshow fciptunnel 17 -c

Displaying tunnel performance

The following example will display performance statistics for a tunnel associated with port 17 on a
Brocade 7800 switch.
switch:admin> portshow fciptunnel 17 --perf

140 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Displaying tunnel TCP statistics

Displaying tunnel TCP statistics

The following example will display TCP connections for a tunnel associated with port 17 on a Brocade
7800 switch.
switch:admin>portshow fciptunnel 17 -c --tcp

You can reset statistics counters to zero to display only new statistics with the --tcp option from the time
you issue the reset using the following command.
switch:admin> portshow fciptunnel 17 -c --tcp --reset

You can display the entire lifetime of statistics for the tunnel using the following command. The time
basis for the statistics will display in the output.
switch:admin> portshow fciptunnel 17 -c --tcp --lifetime

Displaying circuits
The following example will display all circuit information.
switch:admin> portshow fcipcircuit all

Displaying a single circuit

The following example will display information for circuit 1 on tunnel 16 of a Brocade 7800 switch.
switch:admin> portshow fcipcircuit 16 1

Displaying TCP statistics for circuits

The following example displays TCP statistics for circuits associated with VE_Port 12 of an FX8-24
blade.
switch:admin> portshow fcipcircuit 3/12 --tcp

You can reset statistics counters to zero to display only new statistics with the --tcp option from the time
you issue the reset using the following command.
switch:admin> portshow fcipcircuit 3/12 -tcp --reset

You can display the entire lifetime of statistics for the circuit using the following command. The time
basis for the statistics will display in the output.
switch:admin> portshow fcipcircuit 3/12 -tcp --lifetime

Displaying circuit performance

The following example will display circuit performance information for circuit 1 on tunnel 20.
switch:admin> portshow fcipcircuit 20 1 --perf

Displaying QoS prioritization for a circuit

The following example will display QoS prioritization for circuit 1 on tunnel 20 of a Brocade 7800 switch.
switch:admin> portshow fcipcircuit 20 1 --perf --qos

Brocade Fabric OS Extension Administrator's Guide 141

53-1003507-05
Displaying tunnel information (FX8-24 blade)

Displaying tunnel information (FX8-24 blade)

You can use the portShow fcipTunnel command to view the performance statistics and monitor the
behavior of an online tunnel. The command syntax is as follows.
The following example shows portShow fcipTunnel with the --c option to display the circuits of tunnel
0.
switch:admin06> portshow fciptunnel 8/12 0 —c

Tunnel issues
The following are common tunnel issues and recommended actions for you to follow to fix the issue.

Tunnel does not come online

Troubleshoot this issue using the following steps.

1. Confirm Ethernet port is online.
portshow ge1
Eth Mac Address: 00.05.1e.37.93.06
Port State: 1 Online
Port Phys: 6 In_Sync
Port Flags: 0x3 PRESENT ACTIVE
Port Speed: 1G
2. Confirm the IP configuration is correct on both tunnel endpoints using the following command.
portshow ipif ge1
portshow fciptunnel 24 --circuit --config
3. Enter the portCmd --ping command to the remote tunnel endpoint from both endpoints.
The -s value is the source IP address; the -d value is the destination IP address.

portcmd --ping ge1 -s 11.1.1.1 -d 11.1.1.2

If the command is successful, then you have IP connectivity and your tunnel should come up. If not,
continue to the next step.
When using VLANS, VLAN tagging ensures that test traffic traverses the same path as real traffic. A
VLAN tag entry for both the local and remote sides of the route must exist prior to issuing the
portCmd --ping or portCmd --traceroute commands. Refer to Managing the VLAN tag table on
page 105 for details.
4. Enter the portCmd --traceroute command to the remote tunnel endpoint from both endpoints.
portcmd --traceroute ge1 -s 11.1.1.1 -d 11.1.1.2
5. If there are routed IP connections that provide for the tunnel, confirm that both ends of the tunnel
have defined IP routes, and the route gateways are correct. The tunnel or route lookup may fail to
come online because of a missing or incorrect IP route.
Refer to the Configuring an IP route on page 80 to review the setup of the IP route.
6. Confirm the tunnel is configured correctly using the following command.
portshow fciptunnel all

142 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Tunnel goes online and offline

Confirm that the compression, FastWrite, and OSTP settings match at each endpoint or the tunnel
may not come up. Confirm that the local and destination IP address and WWN are accurate.
7. Generate an Ethernet sniffer trace.
Rule out all possible blocking factors. Routers and firewalls that are in the data path must be
configured to pass traffic (TCP port 3225, and in the case of the Brocade 7840 switch, TCP port
3226) and IPsec traffic, if IPsec is used (UDP port 500). If possible blocking factors have been ruled
out, simulate a connection attempt using the portCmd --ping command, from source to destination,
and then generate an Ethernet trace between the two endpoints. The Ethernet trace can be
examined to further troubleshoot the connectivity.

Tunnel goes online and offline

A tunnel that goes offline and then online (bouncing tunnel) is a common problem. This usually occurs
because of an overcommitment of available bandwidth resulting in the following behaviors:
• Too much data tries to go over the link.
• Management data gets lost, queued too long, and timeouts expire.
• Data times out multiple times.
Take the following steps to gather information.
1. Verify what link bandwidth is available.
2. Confirm the IP path is being used exclusively for traffic.
3. Confirm that traffic shaping is configured to limit the available bandwidth using the following
command.
portShow fciptunnel all -tcp

Examine data from both routers. This data shows retransmissions indicating input and output rates
on the tunnels.
4. For the 7800 switch and FX8-24 blade, run Tperf command to gather WAN performance data. For
the 7840 switch, use the WAN tool.

Troubleshooting Extension links

The following list contains information for troubleshooting Extension links:
• When deleting Extension links, you must delete them in the exact reverse order in which they were
created. That is, first delete the tunnels, followed by any IP route configurations, then the IP
interfaces, and finally the port configuration.
• The portCmd --ping command only verifies physical connectivity. This command does not verify that
you have configured the ports correctly for tunnels.
• Ports at both ends of the tunnel must be configured correctly for a tunnel to work correctly. These
ports can be either VE_Ports or VEX_Ports. A VEX_Port must be connected to a VE_Port.
• When configuring routing over an Extension link for a fabric, the edge fabric will use VE_Ports and
the backbone fabric will use VEX_Ports for a single tunnel.
• If a tunnel fails with the "Disabled (Fabric ID Oversubscribed)" message, the solution is to reconfigure
the VEX_Port to the same fabric ID as all of the other ports connecting to the edge fabric.
• Because of an IPsec RASLog limitation, you may not be able to determine an incorrect configuration
that causes an IPsec tunnel to not become active. This misconfiguration can occur on either end of
the tunnel. As a result, you must correctly match the encryption method, authentication algorithm,
and other configurations on each end of the tunnel.

Brocade Fabric OS Extension Administrator's Guide 143

53-1003507-05
Gathering additional information

Gathering additional information

The following commands should be executed and their data collected before the supportsave
command is run. Using the supportsave command can take ten minutes or more to run, and some of
the information is time critical.
• tracedump -n
• porttrace --show all
• porttrace --status
For issues specific to tunnel ports, run and collect the data from the following commands:
• slotshow
• portshow slot/ge_port
For a Brocade 7840 switch, run and collect the data from the following commands:
• extncfg --show
• portcfgge --show
For a Brocade 7840 switch that is running in hybrid mode (for IP Extension features), run and collect
the data from the following commands:
• portshow lag
• portshow lag --detail
• portshow tcl
• portshow tcl --detail
• portshow lan-stats --global
• portshow lan-stats --per-flow
• portshow lan-stats --hist-stats
If possible, run and collect the data from the following commands:
• portshow ipif all slot/ge_port
• portshow arp all slot/ge_port
• portshow iproute all slot/ge_port
• portshow fciptunnel slot/ge_port all|tunnel ID
• portshow fciptunnel all --perf
• portshow fciptunnel all -c
• portshow fciptunnel all --circuit --perf --summary
• portshow fciptunnel all --circuit --perf --tcp --qos
• portCmd --ping --traceroute --perf
• portCmd --ping
• portCmd traceroute
Finally, gather the data from the supportsave command.
Refer to the Fabric OS Administrator's Guide or Fabric OS Command Reference for complete details
on these commands.

Using FTRACE
FTRACE is a support tool used primarily by your switch support provider. FTRACE can be used in a
manner similar to that of a channel protocol analyzer. You can use FTRACE to troubleshoot problems

144 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 FTRACE configuration

through a Telnet session rather than using an analyzer or sending technical support personnel to the
installation site.

CAUTION

FTRACE is meant to be used solely as a support tool and should be used only by Brocade
support personnel, or at the request of Brocade support personnel. The FTRACE command is
restricted to the root switch user.

FTRACE is always enabled on extension switches and blades, and the trace data is automatically
captured.

FTRACE configuration
A default configuration for FTRACE is provided for each of the two DP complexes on the Brocade
FX8-24 blade and 7840 switch and for the single Brocade 7800 FCIP DP complex. This allows tracing
of events related to the DP complexes.
You can use the root portcfg ftrace slot/ge_port cfg command to change FTRACE configuration
settings as described in Configuring IP addresses and routes on page 124.

Brocade 7800 switch and FX8-24 blade

The default configuration creates four FTRACE buffers of 100,000 trace events that will be used until a
trigger event (programmed trigger point in the logic) occurs. Trigger events include unexpected events
or events that include FC abort sequences or other errors when emulation features are enabled on the
tunnel.
The default configuration does not allow reuse of a trace buffer that includes one or more trigger events.
The FTRACE configuration item that controls this function is called Auto Checkout (ACO). The default
configuration of FTRACE provides for capturing, at a minimum, the first four error time periods in the
four FTRACE buffers. That is because the default setting has enabled FTRACE ACO processing. When
a buffer is checked out, it will not be reused until it is manually checked in or cleared through the
supportsave process.
If the FTRACE configuration is changed so that ACO is disabled, then instead of post-filling and then
checking out, the buffer is marked as triggered. If multiple trigger events subsequently occur so that all
buffers are marked triggered, FTRACE will find the oldest triggered buffer and make it the current buffer.
In this configuration, FTRACE will be set up to capture the last three error time periods.
FTRACE data contents are included in a switch supportsave capture. After the supportsave has been
captured, the FTRACE buffers will be reset and all buffers that were previously either "checked out" or
"triggered" return to an "unused" state.
Change the FTRACE ACO configuration using the following root command:

portcfg ftrace [slot/]vePort cfg

Refer to Changing configuration settings on page 146 for more information.

Brocade 7840 switch

FTRACE has been enhanced on the Brocade 7840 to allow more trace saving options than for the
Brocade 7800 switch or FX8-24 blade. The default FTRACE configuration has been changed on this
platform as a result of those enhancements. For a display of the default configuration for the Brocade

Brocade Fabric OS Extension Administrator's Guide 145

53-1003507-05
Changing configuration settings

7840 using the portshow ftrace ve_port stats command, refer to Displaying FTRACE status on 7840
switch on page 149.
The Brocade 7840 includes two Data Processing (DP) complexes. Each DP complex has an FTRACE
instance. The default configuration for FTRACE on the Brocade 7840 defines eight FTRACE buffers
for trace events on each DP complex. The default configuration defines 300,000 trace entries (trace
records) per trace buffer. The default FTRACE configuration enables auto checkout (ACO) for the first
four buffers and disables ACO for the last four. The Brocade 7840 switch has a solid state disk (SSD)
file system in each DP complex. This can be used to save copies of triggered FTRACE buffers. Use of
the SSD to save FTRACE buffers is enabled by default and by the "Save to Flash" portcfg ftrace
ve_port cfg command.
On the Brocade 7840, you can enable ACO for each defined FTRACE buffer. FTRACE processing
varies when the FTRACE buffer is defined with ACO enabled or disabled.
ACO enabled - If the FTRACE buffer is defined with ACO enabled, when that buffer is the "current"
FTRACE buffer and a trigger event occurs, FTRACE will post fill that buffer to the end (or add the post
fill percentage of more trace entries). When the post filling process is occurring the FTRACE buffer
state will be reported as "post fill". When the post filling process has completed, the buffer state will be
reported as "checked out," and the next sequential available buffer number will be assigned to the
current buffer (state "current"). If all FTRACE buffers are marked as "checked out," FTRACE will no
longer be recording trace entries. The default configuration therefore will capture at least the first four
error traces, permanently check out those buffers, and then move them to the ACO-off buffers.
FTRACE buffers that have been checked out will be saved in a supportsave capture. When the
supportsave is complete, the buffers will return to an "unused" state and will be available for new
traces. You can use the portshow ftrace ve_port cmd command to check in a checked out buffer.
ACO disabled - If the FTRACE buffer is defined with ACO disabled, when that buffer is the "current"
FTRACE buffer and a trigger event occurs, FTRACE processing will complete the same post filling
process as described above. When completed, if the "Save to Flash" configuration option was
enabled, the buffer will move to a "saving" state, and the next available buffer will be made as the
current trace buffer. The Brocade 7840 will save as many as eight FTRACE buffers in the DP SSD file
system. If there are already eight saved FTRACE buffers in the file system, the oldest trace buffer will
be replaced by the current buffer being saved. When the save-to-flash processing completes, the
buffer will be marked as "triggered". If the "Save to Flash" option is not enabled, the buffer will be
immediately marked as "triggered" and the next sequentially available FTRACE buffer will be marked
as the "Current" buffer.
In the default configuration, FTRACE will therefore capture at least the first four error events (in buffers
1, 2, 3 and 4). It will capture the last three error events in triggered buffers (5-7) and will always have a
current buffer. Buffers 5-7 will also potentially have as many as 10 saved prior trigger events reported
and saved in the DP SSD file system.
FTRACE data contents are included in a switch supportsave capture. After the supportsave has been
captured, the FTRACE buffers will be reset and all buffers that were previously either "Checked Out"
or "Triggered" return to an "unused" state.
Change the FTRACE ACO configuration using the root portcfg ftrace ve_port cfg command. Refer to
Changing configuration settings on page 146 for more information.

Changing configuration settings

Use the root portcfg ftrace slot/ge_port cfg command to change FTRACE configuration settings. The
configuration for FTRACE is defined using the first VE_Port on the switch or blade DP complex as
follows:
• Brocade 7800 switch - VE_Port 16
• Brocade FX8-24 blade - VE_Port 22 on DP0 and VE_Port 12 on DP1
• Brocade 7840 switch - VE_Port 24 on DP0 and VE_Port 34 on DP1

146 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Brocade 7840 switch example

To change FTRACE configuration settings on a Brocade 7800 switch, if applicable, set the context
where VE port 16 is defined, and then issue the following command as root user only:
portcfg ftrace 16 cfg

To change FTRACE configuration settings on the first DP complex (DP0) on a Brocade 7840 switch, if
applicable, set the context where VE_ Port 24 is defined, and then issue the following command as the
root user only:

portcfg ftrace 24 cfg

To change FTRACE configuration settings on the first DP complex (DP0) on a Brocade FX8-24 blade, if
applicable, set the context where the VE_Port 22 is defined, and then issue the following command as
the root user only:

portcfg ftrace slot_number/22 cfg

To change FTRACE configuration settings on the second DP complex on a Brocade FX8-24 (DP1), if
applicable, set the context to where VE port 12 is defined, and then issue the following command as the
root user only:

portcfg ftrace slot_number/12 cfg

To change FTRACE configuration settings on the first DP complex (DP0) on a Brocade 7840 switch, if
applicable, set the context where the VE_Port 24 is defined, and then issue the following command as
the root user only:

portcfg ftrace 24 cfg

To change FTRACE configuration settings on the second DP complex (DP1) on a 7840 switch, if
applicable, set the context to where VE port 34 is defined, and then issue the following command as the
root user only:

portcfg ftrace 34 cfg

Note that portcfg is an interactive command sequence and will prompt you for configuration items.

Brocade 7840 switch example

Following is an example of the interactive command sequence that illustrates where you are prompted
to change FTRACE configuration settings on a Brocade 7840 switch. To change the settings, set the
context where VE_Port 34 is defined, and then issue the portcfg ftrace 34 cfg command as root user
only.

NOTE
User input lines in following example of this interactive command have been annotated to help you
select configuration options. Those notes in italic font, such as *Enables FTRACE (default is y)*,
indicate options that you can modify. Those in bold font, such as as *Sets the trace mask*, indicate
options that you should not modify without direction from a support representative.
switch_10:FID10:root> portcfg ftrace 34 cfg

*** FTRACE INTERACTIVE CONFIGURATION ***

*** Note: A reboot is necessary to ***

*** activate a change in the number ***
*** of buffers or records. ***
Enable FTRACE? (Y,y,N,n): [y] y *Enables FTRACE -default y*
Buffers (0-16): [8] *Sets number of trace buffers -default 8*
Records (decimal, no commas) (0-262,144): [300,000] *Sets number of trace records per buffer
-default 200,000*
Auto Checkout? (Y,y,N,n): [y] *Enables ACO (default y)*

Brocade Fabric OS Extension Administrator's Guide 147

53-1003507-05
Brocade 7800 switch example

Auto Checkout is on, config at least 1 buffer accordingly.

Auto Checkout buffer 0 (Y,y,N,n): [y] *Enables ACO for buffer 0 -default y*
Auto Checkout buffer 1 (Y,y,N,n): [y] *Enables ACO for buffer 1 -default y*
Auto Checkout buffer 2 (Y,y,N,n): [y] *Enables ACO for buffer 2 -default y*
Auto Checkout buffer 3 (Y,y,N,n): [y] *Enables ACO for buffer 3 -default y*
Auto Checkout buffer 4 (Y,y,N,n): [n] *Disables ACO for buffer 4 -default n*
Auto Checkout buffer 5 (Y,y,N,n): [n] *Disables ACO for buffer 5 -default n*
Auto Checkout buffer 6 (Y,y,N,n): [n] *Disables ACO for buffer 6 -default n*
Auto Checkout buffer 7 (Y,y,N,n): [n] *Disables ACO for buffer 7 -default n*
Save to Flash? (Y,y,N,n): [y] *Enables saving non-ACO to flash -
default y*
Post Percentage (decimal) (0-100): [5] *Sets the post fill percentage -default
5*
Trace Mask (*) (0-ffffffff): [8000dffb] *Sets the trace mask -default 8000dffb*
Trigger Mask (T) (0-ffffffff): [1] *Sets the trigger mask -default 1*
Display Mask (-) (0-ffffffff): [ffffffff] *Sets the trace display mask -default
ffffffff*
Enable VE Traces? (Y,y,N,n): [y] *Enables VE event traces -default y*
Enable FCIP Traces? (Y,y,N,n): [y] *Enables FCIP event traces -default y*
Enable TCPIP Traces? (Y,y,N,n): [y] *Enables TCP/IP event traces -default y*
Enable TCPIP Conn Traces? (Y,y,N,n): [y] *Enables TCP/IP Connection event traces -
default y*
Enable IP Traces? (Y,y,N,n): [y] *Enables IP Event traces -default y*
Enable ARL Traces? (Y,y,N,n): [y] *Enables ARL Event traces -default y*
Enable Ethernet Traces? (Y,y,N,n): [n] *Disables Ethernet traces -default n*
Enable IP API Traces? (Y,y,N,n): [y] *Enables IP/API even traces -default y*
Enable FCIP MSG Traces? (Y,y,N,n): [y] *Enables FCIP Msg traces -default y*
Enable VDM Traces? (Y,y,N,n): [n] *Disables VDM traces -default n*
Configuration complete.
Operation Succeeded
switch_10:FID10:root>

To correctly and completely delete an FTRACE configuration and reset to defaults, perform the
following command sequences.

switch_10:FID10:root> portcfg ftrace 34 del

*** Note: This command will clear out ***

*** the current config and FTRACE will ***
*** be reset to default values. ***

Do you wish to continue? (Y,y,N,n): [n] y

Operation Successful

switch_10:FID10:root> reboot
/* After switch completes reboot sequence */
switch_10:FID10:root> portcfg ftrace 34 cfg
/* repeat the configuration or leave as default */

Brocade 7800 switch example

Following is an example of the interactive command sequence that illustrates where you are prompted
to change FTRACE configuration settings on a Brocade 7800 switch. To change the settings, set the
context where VE_Port 16 is defined, and then issue the portcfg ftrace 16 cfg command as root user
only.

switch6:root> portcfg ftrace 16 cfg

*** FTRACE INTERACTIVE CONFIGURATION ***
*** Note: A reboot is necessary to ***
*** activate a change in the number ***
*** of buffers or records. ***
Setting up ftrace configuration defaults.
Enable FTRACE? (Y,y,N,n): [n] y
Auto Checkout? (Y,y,N,n): [n] y or n
Buffers (0-6): [4] 6
Records (decimal, no commas) (0-349,520): [100,000] 120000
Post Percentage (decimal) (0-100): [5] 6
Reference the table below to set the TRACE, TRIGGER, and DISPLAY masks
*-Bit 31 [0x80000000]: Software Structure
-Bit 19 [0x00080000]: EtRX - Ethernet Received Frame

148 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Displaying FTRACE status on a DP complex

-Bit 18 [0x00040000]: EtSX - Ethernet Send Frame to Peer

-Bit 17 [0x00020000]: TnTX - Tunnel Received Peer Frame
-Bit 16 [0x00010000]: TnSX - Tunnel Send Frame to Peer
-Bit 15 [0x00008000]: FcT - FC FWD Frame From Peer
-Bit 14 [0x00004000]: FcR - FC FWD Received Frame
-Bit 13 [0x00002000]: Dsc - Discarded Frame
-Bit 12 [0x00001000]: Data - Frame Data
*-Bit 11 [0x00000800]: State Change
*-Bit 10 [0x00000400]: CpRX - Frame Received From CP
-Bit 9 [0x00000200]: CpSX - Frame Sent To CP
*-Bit 8 [0x00000100]: ToP - Sent To Peer
-Bit 7 [0x00000080]: Tfx - Emulation FC Frame From Peer
*-Bit 6 [0x00000040]: Rfx - Emulation FC Received Frame
*-Bit 5 [0x00000020]: Sfx - Send Frame
*-Bit 4 [0x00000010]: Gfx - Generated Frame
*-Bit 3 [0x00000008]: FC SOFi1/2/3 or Class F Frames
-Bit 2 [0x00000004]: FC SOFn1/2/3 Frames
T*-Bit 1 [0x00000002]: Msg - Information
T*-Bit 0 [0x00000001]: Err - Error
Trace Mask (*) (0-ffffffff): [8000fefb]
Trigger Mask (T) (0-ffffffff): [1]
Display Mask (-) (0-ffffffff): [8000fefb]
Enable VE Traces? (Y,y,N,n): [y]
Enable FCIP Tunnel Traces? (Y,y,N,n): [y]
Enable TCPIP Traces? (Y,y,N,n): [y]
Enable TCPIP Conn Traces? (Y,y,N,n): [n]
Enable IP Traces? (Y,y,N,n): [n]
Enable ARL Traces? (Y,y,N,n): [n]
Enable Ethernet Traces? (Y,y,N,n): [n]
Enable IP API Traces? (Y,y,N,n): [n]
Enable FCIP MSG Traces? (Y,y,N,n): [n]
Enable VDM traces? (Y,y,N,n): [n]
Operation Succeeded
spike64:root>

Displaying FTRACE status on a DP complex

To display the current FTRACE status on an DP complex, issue the following command as the root
user:

portshow ftrace [slot/]vePort stats

The vePort is in the current logical switch context.

Brocade 7840 switch example

Following is an example of displaying FTRACE status using the portshow ftrace slot/ve_port stats
command. Note that this is the default configuration for the Brocade 7840 switch.
switch:admin> portshow ftrace 34 stats

VE traces: On-all Trace Mask: 0x8000dffb (*)

FCIP Tunnel traces: On-all Trigger Mask: 0x00000001 (T)
TCPIP traces: On-all Display Mask: 0xffffffff (-)
TCPIP Conn. traces: On-all Tunnel Mask: Inactive
IP traces: On-all Post trigger: 5% - 10000 events
ARL traces: On-all Record Size: 128
ETHERNET traces: Off Save to Flash: Enabled
IP API traces: On-all FTRACE is: Enabled
FCIP MSG traces: On-all Debug level: 4-Normal (low)
VDM traces: Off CLIB / HAL: Off / Off

*-Bit 31 [0x80000000]: Software Structure

-Bit 19 [0x00080000]: EtRX - Ethernet Received Frame
-Bit 18 [0x00040000]: EtSX - Ethernet Send Frame to Peer
-Bit 17 [0x00020000]: TnTX - Tunnel Received Peer Frame
-Bit 16 [0x00010000]: TnSX - Tunnel Send Frame to Peer
*-Bit 15 [0x00008000]: FcT - FC FWD Frame From Peer
*-Bit 14 [0x00004000]: FcR - FC FWD Received Frame
-Bit 13 [0x00002000]: Dsc - Discarded Frame

Brocade Fabric OS Extension Administrator's Guide 149

53-1003507-05
Brocade 7800 switch or FX8-24 blade example

*-Bit 12 [0x00001000]: Data - Frame Data

*-Bit 11 [0x00000800]: State Change
*-Bit 10 [0x00000400]: CpRX - Frame Received From CP
*-Bit 9 [0x00000200]: CpSX - Frame Sent To CP
*-Bit 8 [0x00000100]: ToP - Sent To Peer
*-Bit 7 [0x00000080]: Tfx - Emulation FC Frame From Peer
*-Bit 6 [0x00000040]: Rfx - Emulation FC Received Frame
*-Bit 5 [0x00000020]: Sfx - Send Frame
*-Bit 4 [0x00000010]: Gfx - Generated Frame
*-Bit 3 [0x00000008]: FC SOFi1/2/3 or Class F Frames
-Bit 2 [0x00000004]: FC SOFn1/2/3 Frames
*-Bit 1 [0x00000002]: Msg - Information
T*-Bit 0 [0x00000001]: Err - Error

+-----+----------+-----+--------+------------+-----+------+------+-------+--------+
| | | | |Trace Header|Wrap | In | Out |Switch | Switch |
| Id | State | ACO | Size | Address |Count| OXID | OXID | Date | Time |
+-----+----------+-----+--------+------------+-----+------+------+-------+--------+
| 0 | Current | on | 200000 | 0x0b0f7480 | 0 | FFFF | FFFF | | |
| 1 | unused | on | 200000 | 0x0b0f7780 | 0 | FFFF | FFFF | | |
| 2 | unused | on | 200000 | 0x0b0f7a80 | 0 | FFFF | FFFF | | |
| 3 | unused | on | 200000 | 0x0b0f7d80 | 0 | FFFF | FFFF | | |
| 4 | unused | off | 200000 | 0x0b0f8080 | 0 | FFFF | FFFF | | |
| 5 | unused | off | 200000 | 0x0b0f8380 | 0 | FFFF | FFFF | | |
| 6 | unused | off | 200000 | 0x0b0f8680 | 0 | FFFF | FFFF | | |
| 7 | unused | off | 200000 | 0x0b0f8980 | 0 | FFFF | FFFF | | |
+-----+----------+-----+--------+------------+-----+------+------+-------+--------+

The table at the bottom of the output example has the following columns:
• Id - The FTRACE trace buffer identifier or buffer number.
• State - The FTRACE buffer state for that buffer number. The state can be one of the following:
‐ Current - The buffer is the current active buffer in use for events.
‐ Triggered - The buffer has been used to record an error event from the DP complex. This state is
used only when the Auto Checkout option was disabled.
‐ Checked Out - The buffer has been used to record an error event from the DP complex, and the
buffer will not be overwritten.
‐ Post Fill - A trigger event has been encountered, and the FTRACE buffer is currently being post-
filled with a number of post-error events. Once the post-filling has been completed, the buffer will
transition to either a "Checked Out" or "Triggered" state.
‐ Unused -The buffer has not been used to capture any events. The buffer will be used when the
prior buffer in the list transitions to either a "Checked Out" or "Triggered" state.
• ACO - Auto Checkout enabled (on) or disabled (off) status.
• Size - The number of trace records that are in the buffer.
• Trace Header Address - A memory address used internally for controlling access to the trace buffer.
• Wrap Count - The number of times that a trace buffer has been wrapped. The trace is a circular
buffer that wraps after the size number of trace events has been exceeded.
• In OXID and Out OXID - Not used until the buffer is being analyzed.
• Switch Date - Indicates the system date when the buffer transitioned to either a "Checked Out" or
"Triggered" state.

Brocade 7800 switch or FX8-24 blade example

Following is an example of displaying FTRACE status using the portshowftrace slot/ve_port stats
command.
Slot 0:
VE traces (0-31): (0xffffffff) On Trace Mask: 0x8000fefb (*)
FCIP Tunnel traces (32-64): On Trigger Mask: 0x00000001 (T)
TCPIP traces (65): On Display Mask: 0x8000fefb (-)
TCPIP Conn. traces (66): Off Tunnel Mask: Inactive
IP traces (67-83): Off Post trigger: 3% - 3600 events
ARL traces (84): Off Record Size: 128
ETHERNET traces (85-103): Off Auto Checkout: Enabled
IP API traces (104): Off FTRACE is: Enabled
FCIP MSG traces (105): Off Debug level: 4-Normal (low)

150 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05
 Management and Troubleshooting

VDM traces (106): Off

*-Bit 31 [0x80000000]: Software Structure
Bit 19 [0x00080000]: EtRX - Ethernet Received Frame
Bit 18 [0x00040000]: EtSX - Ethernet Send Frame to Peer
Bit 17 [0x00020000]: TnTX - Tunnel Received Peer Frame
Bit 16 [0x00010000]: TnSX - Tunnel Send Frame to Peer
*-Bit 15 [0x00008000]: FcT - FC FWD Frame From Peer
*-Bit 14 [0x00004000]: FcR - FC FWD Received Frame
*-Bit 13 [0x00002000]: Dsc - Discarded Frame
*-Bit 12 [0x00001000]: Data - Frame Data
*-Bit 11 [0x00000800]: State Change
*-Bit 10 [0x00000400]: CpRX - Frame Received From CP
*-Bit 9 [0x00000200]: CpSX - Frame Sent To CP
Bit 8 [0x00000100]: ToP - Sent To Peer
*-Bit 7 [0x00000080]: Tfx - Emulation FC Frame From Peer
*-Bit 6 [0x00000040]: Rfx - Emulation FC Received Frame
*-Bit 5 [0x00000020]: Sfx - Send Frame
*-Bit 4 [0x00000010]: Gfx - Generated Frame
*-Bit 3 [0x00000008]: FC SOFi1/2/3 or Class F Frames
Bit 2 [0x00000004]: FC SOFn1/2/3 Frames
*-Bit 1 [0x00000002]: Msg - Information
T*-Bit 0 [0x00000001]: Err - Error
+-----+----------+--------+------------+-------+------+------+--------+--------+
| | | |Trace Header| Wrap | In | Out | Switch | Switch |
| Id | State | Size | Address | Count | OXID | OXID | Date | Time |
+-----+----------+--------+------------+-------+------+------+--------+--------+
| 1 | Current | 100000 | 0x001f2f00 | 12344 | FFFF | FFFF | | |
| 1 | unused | 100000 | 0x001f3180 | 0 | FFFF | FFFF | | |
| 2 | unused | 100000 | 0x001f3400 | 0 | FFFF | FFFF | | |
| 3 | unused | 100000 | 0x001f3680 | 0 | FFFF | FFFF | | |
+-----+----------+--------+------------+-------+------+------+--------+--------+

The table at the bottom of the output example has the following information:
• Id — The FTRACE trace buffer identifier or buffer number.
• State — The FTRACE buffer state for that buffer number. The state can be one of the following:
‐ Current — The buffer is the current active buffer in use for events
‐ Triggered — The buffer has been used to record an error event from the DP complex. This state is
used only when the Auto Checkout option was disabled.
‐ Checked Out — The buffer has been used to record an error event from the DP complex, and the
buffer will not be overwritten.
‐ Post Fill — A trigger event has been encountered, and the FTRACE buffer is currently being post-
filled with a number of post-error events. Once the post-filling has been completed, the buffer will
transition to either a “Checked Out” or “Triggered” state.
‐ Unused — The buffer has not been used to capture any events. The buffer will be used when the
prior buffer in the list transitions to either a “Checked Out” or “Triggered” state.
• Size — The number of trace records that are in the buffer.
• Trace Header Address — A memory address used internally for controlling access to the trace buffer.
• Wrap Count — The number of times that a trace buffer has been wrapped. The trace buffer is a
circular buffer that wraps after the size number of trace events has been exceeded.
• In OXID and Out OXID — Not used until the buffer is being analyzed.
• Switch Date — Indicates the system date when the buffer transitioned to either a “Checked Out” or
“Triggered” state.
• Switch Time — Indicates the system time when the buffer transitioned to either a “Checked Out” or
“Triggered” state.

Brocade Fabric OS Extension Administrator's Guide 151

53-1003507-05
Brocade 7800 switch or FX8-24 blade example

152 Brocade Fabric OS Extension Administrator's Guide

53-1003507-05