Updated for 8.3.

Clustered Data ONTAP 8.3

MetroCluster Installation and Configuration Guide

NetApp, Inc. Telephone: +1 (408) 822-6000 Part number: 215-10107_B0

495 East Java Drive Fax: +1 (408) 822-4501 September 2015
Sunnyvale, CA 94089 Support telephone: +1 (888) 463-8277
U.S. Web: www.netapp.com
Feedback: [email protected]
 Table of Contents | 3

Contents
MetroCluster documentation ...................................................................... 8
Preparing for the MetroCluster installation ............................................ 10
Differences between 7-Mode and clustered Data ONTAP MetroCluster
configurations ...................................................................................................... 10
Differences between the clustered Data ONTAP MetroCluster configurations ........ 11
Considerations for configuring cluster peering ......................................................... 12
Prerequisites for cluster peering .................................................................... 12
Considerations when using dedicated ports .................................................. 13
Considerations when sharing data ports ........................................................ 13
Considerations for MetroCluster configurations with native disk shelves or
array LUNs .......................................................................................................... 14
Considerations when transitioning from 7-Mode to clustered Data ONTAP ........... 14
Configuration of new MetroCluster systems ............................................................. 15
Preconfigured component passwords ............................................................ 15
Hardware setup checklist .............................................................................. 15
Software setup checklist ................................................................................ 17
Choosing the correct installation procedure for your configuration ..... 20
Cabling a four-node or two-node fabric-attached MetroCluster
configuration .......................................................................................... 22
Parts of a fabric MetroCluster configuration ............................................................. 23
Local HA pairs .............................................................................................. 27
Redundant FC-to-SAS bridges ...................................................................... 27
Redundant FC switch fabrics ........................................................................ 28
The cluster peering network .......................................................................... 28
Required MetroCluster components and naming guidelines for fabric
configurations ...................................................................................................... 29
Worksheet for FC switches and FC-to-SAS bridges ................................................. 31
Installing and cabling MetroCluster components ...................................................... 33
Racking the hardware components ............................................................... 33
Cabling the HBA adapters to the FC switches .............................................. 34
Cabling the ISLs between MetroCluster sites ............................................... 37
Recommended port assignments for FC switches ......................................... 38
Cabling the cluster interconnect in four-node configurations ....................... 41
Cabling the cluster peering connections ........................................................ 41
Cabling the HA interconnect, if necessary .................................................... 42
Cabling the management and data connections ............................................ 43
Installing FC-to-SAS bridges and SAS disk shelves ................................................ 43
Preparing for the installation ......................................................................... 44
Installing the FC-to-SAS bridge and SAS shelves ........................................ 45
Configuring the FC switches ..................................................................................... 49
Configuring the FC switches by running a configuration file ....................... 49
4 | MetroCluster Installation and Configuration Guide

Configuring the Cisco or Brocade FC switches manually ............................ 50

Cabling a two-node SAS-attached stretch MetroCluster
configuration .......................................................................................... 87
Parts of a two-node direct-attached MetroCluster configuration .............................. 87
Required MetroCluster components and naming guidelines for two-node stretch
configurations ...................................................................................................... 88
Installing and cabling MetroCluster components ...................................................... 89
Racking the hardware components ............................................................... 90
Cabling the controllers to each other and the storage shelves ....................... 91
Cabling the cluster peering connections ........................................................ 92
Cabling the management and data connections ............................................ 92
Cabling a two-node bridge-attached stretch MetroCluster
configuration .......................................................................................... 94
Required MetroCluster components and naming guidelines for two-node stretch
configurations ...................................................................................................... 94
Parts of a two-node MetroCluster configuration with FC-to-SAS bridges ............... 95
Gathering required information and reviewing the workflow ................................... 96
Worksheet for FC-to-SAS bridges ................................................................ 97
Installing and cabling MetroCluster components ...................................................... 98
Racking the hardware components ............................................................... 98
Cabling the controllers to each other ............................................................. 99
Cabling the cluster peering connections ........................................................ 99
Cabling the management and data connections ............................................ 99
Installing FC-to-SAS bridges and SAS disk shelves .............................................. 100
Preparing for the installation ....................................................................... 101
Installing the FC-to-SAS bridge and SAS shelves in a fabric-attached
configuration .......................................................................................... 102
Cabling the FC-to-SAS bridges to the controller module in a two-node
direct-attached configuration ................................................................. 106
Configuring hardware for sharing a Brocade 6510 FC fabric during
transition .............................................................................................. 107
Reviewing Brocade license requirements ............................................................... 108
Racking the hardware components ......................................................................... 108
Cabling the new MetroCluster controllers to the existing FC fabrics ..................... 109
Configuring switch fabrics sharing between the 7-Mode and clustered
MetroCluster configuration ............................................................................... 110
Disabling one of the switch fabrics ............................................................. 110
Deleting TI zoning and configuring IOD settings ....................................... 112
Ensuring ISLs are in the same port group and configuring zoning ............. 113
Reenabling the switch fabric and verify operation ...................................... 114
Configuring the MetroCluster software in Data ONTAP ..................... 116
Gathering required information and reviewing the workflow ................................. 117
IP network information worksheet for site A .............................................. 117
IP network information worksheet for site B .............................................. 119
 Table of Contents | 5

Similarities and differences between regular cluster and MetroCluster

configurations .................................................................................................... 122
Setting a previously used controller module to system defaults in Maintenance
mode .................................................................................................................. 122
Verifying disk assignment in Maintenance mode in a four-node configuration ..... 123
Assigning disk ownership in non-AFF systems .......................................... 125
Assigning disk ownership in AFF systems ................................................. 126
Verifying disk assignment in Maintenance mode in a two-node configuration ...... 128
Verifying the HA state of components is mcc or mcc-2n in Maintenance mode .... 129
Setting up Data ONTAP .......................................................................................... 130
Running System Setup in a four-node MetroCluster configuration ............ 130
Setting up the clusters in a two-node MetroCluster configuration .............. 133
Configuring the clusters into a MetroCluster configuration ................................... 135
Peering the clusters ...................................................................................... 135
Mirroring the root aggregates ...................................................................... 145
Creating a mirrored data aggregate on each node ....................................... 145
Implementing the MetroCluster configuration ............................................ 146
Configuring the MetroCluster FC switches for health monitoring ............. 148
Checking the MetroCluster configuration ................................................... 149
Checking for MetroCluster configuration errors with Config Advisor ................... 151
Verifying local HA operation .................................................................................. 151
Verifying switchover, healing, and switchback ....................................................... 153
Installing the MetroCluster Tiebreaker software ..................................................... 153
Protecting configuration backup files ...................................................................... 153
Planning and installing a MetroCluster configuration with array
LUNs ..................................................................................................... 154
Planning for a MetroCluster configuration with array LUNs ................................. 154
Supported MetroCluster configuration with array LUNs ........................................ 154
Requirements for a MetroCluster configuration with array LUNs ......................... 155
Installing and cabling the MetroCluster components in a configuration with
array LUNs ........................................................................................................ 156
Racking the hardware components in a MetroCluster configuration with
array LUNs ............................................................................................ 157
Preparing a storage array for use with Data ONTAP systems .................... 157
Switch ports required for a MetroCluster configuration with array LUNs . 158
Cabling the FC-VI and HBA ports in a MetroCluster configuration with
array LUNs ............................................................................................ 159
Cabling the ISLs between MetroCluster sites ............................................. 164
Cabling the cluster interconnect in four-node configurations ..................... 165
Cabling the cluster peering connections ...................................................... 166
Cabling the HA interconnect, if necessary .................................................. 166
Cabling the management and data connections .......................................... 167
Cabling storage arrays to FC switches in a MetroCluster configuration .... 167
Switch zoning in a MetroCluster configuration with array LUNs .......................... 171
6 | MetroCluster Installation and Configuration Guide

Requirements for switch zoning in a MetroCluster configuration with

array LUNs ............................................................................................ 171
Example of switch zoning in a two-node MetroCluster configuration
with array LUNs .................................................................................... 172
Example of switch zoning in a four-node MetroCluster configuration
with array LUNs .................................................................................... 173
Setting up Data ONTAP in a MetroCluster configuration with array LUNs .......... 175
Verifying the HA state of components is mcc or mcc-2n in Maintenance
mode ...................................................................................................... 176
Configuring Data ONTAP on a system that uses only array LUNs ............ 177
Setting up the cluster ................................................................................... 179
Installing the license for using array LUNs in a MetroCluster
configuration .......................................................................................... 180
Assigning ownership of array LUNs ........................................................... 180
Peering the clusters ...................................................................................... 181
Mirroring the root aggregates ...................................................................... 183
Creating a mirrored data aggregate on each node ....................................... 184
Implementing the MetroCluster configuration ............................................ 185
Configuring the MetroCluster FC switches for health monitoring ............. 187
Checking the MetroCluster configuration ................................................... 188
Implementing a MetroCluster configuration with both disks and array LUNs ....... 190
Considerations when implementing a MetroCluster configuration with
disks and array LUNs ............................................................................ 190
Examples of two-node stretch MetroCluster configurations with disks
and array LUNs ..................................................................................... 191
Example of a two-node fabric-attached MetroCluster configuration with
disks and array LUNs ............................................................................ 193
Example of a four-node MetroCluster configuration with disks and array
LUNs ..................................................................................................... 194
Using the OnCommand management tools for further configuration
and monitoring ..................................................................................... 196
Synchronizing the system time using NTP ............................................................. 196
Requirements and limitations when using Data ONTAP in a
MetroCluster configuration ................................................................ 198
Cluster peering from the MetroCluster sites to a third cluster ................................ 198
Volume creation on a root aggregate ....................................................................... 198
Networking and LIF creation guidelines for MetroCluster configurations ............. 198
Volume or FlexClone command VLDB errors ........................................................ 200
Output for storage disk show and storage shelf show commands in a two-node
MetroCluster configuration ............................................................................... 200
Modifying volumes to set NVFAIL in case of switchover ...................................... 200
Monitoring and protecting database validity by using NVFAIL ............................. 200
How NVFAIL protects database files .......................................................... 201
Commands for monitoring data loss events ................................................ 201
Accessing volumes in NVFAIL state after a switchover ............................. 202
 Table of Contents | 7

Recovering LUNs in NVFAIL states after switchover ................................ 202

Glossary of MetroCluster terms ............................................................. 203
Copyright information ............................................................................. 205
Trademark information ........................................................................... 206
How to send comments about documentation and receive update
notifications .......................................................................................... 207
Index ........................................................................................................... 208
8

MetroCluster documentation
There are a number of documents that can help you configure, operate, and monitor a MetroCluster
configuration.

MetroCluster and Data ONTAP libraries

Library Content
NetApp Documentation: MetroCluster in All MetroCluster guides
clustered Data ONTAP
NetApp Documentation: Clustered Data All Data ONTAP express guides
ONTAP Express Guides
NetApp Documentation: Data ONTAP 8 All Data ONTAP guides
(current releases)

MetroCluster and miscellaneous guides

Guide Content
NetApp Technical Report 4375: MetroCluster A technical overview of the MetroCluster
for Data ONTAP Version 8.3 Overview and configuration and operation.
Best Practices
Best practices for MetroCluster
configuration.

Clustered Data ONTAP 8.3 MetroCluster How to install a MetroCluster system that has
Installation Express Guide been received from the factory. You should use
this guide only if the following is true:

The MetroCluster configuration has been

received from the factory.

The configuration is using Brocade FC

storage switches.
This guide does not document configuration
of the Cisco FC storage switches.
The configuration is not using array LUNs
(FlexArray Virtualization).

The configuration is not sharing existing FC

fabrics with a 7-Mode fabric MetroCluster
during transition.

Clustered Data ONTAP 8.3 MetroCluster Understanding the MetroCluster

Management and Disaster Recovery Guide configuration

Switchover, healing and switchback

Disaster recovery
 MetroCluster documentation | 9

Guide Content
MetroCluster Service Guide Guidelines for maintenance in a
MetroCluster configuration

Hardware replacement and firmware

upgrade procedures for FC-to-SAS bridges
and FC switches

Hot-adding a disk shelf

Hot-removing a disk shelf

Replacing hardware at a disaster site

MetroCluster Tiebreaker Software Installation Monitoring the MetroCluster configuration

and Configuration Guide with the MetroCluster Tiebreaker software

Clustered Data ONTAP 8.3 Data Protection How mirrored aggregates work
Guide
SyncMirror

SnapMirror

SnapVault

NetApp Documentation: OnCommand Unified Monitoring the MetroCluster configuration

Manager Core Package (current releases)
NetApp Documentation: OnCommand Monitoring MetroCluster performance
Performance Manager for Clustered Data
ONTAP
7-Mode Transition Tool 2.1 Data and Transitioning data from 7-Mode storage
Configuration Transition Guide systems to clustered storage systems
10

Preparing for the MetroCluster installation

As you prepare for the MetroCluster installation, you should understand the MetroCluster hardware
architecture and required components. If you are familiar with MetroCluster configurations in a 7-
mode environment, you should understand the key MetroCluster differences you find in a clustered
Data ONTAP environment.

Differences between 7-Mode and clustered Data ONTAP

MetroCluster configurations
There are key differences between clustered Data ONTAP MetroCluster configurations and
configurations with Data ONTAP operating in 7-Mode.
In clustered Data ONTAP, a four-node MetroCluster configuration includes two HA pairs, each in a
separate cluster at physically separated sites.

Feature or Clustered Data ONTAP Data ONTAP 7-Mode

component
Four node Two node
Number of Four Two, one at each site Two
storage The controllers are Each controller is The two controllers are
controllers configured as two HA configured as a single- configured as a HA pair
pairs, one HA pair at node cluster. with one controller at each
each site. site.
Local failover Yes No No
available? A failover can occur In the event of a problem In the event of a problem
at either site without at the local site, the at the local site, the system
triggering an overall system switches over to fails over to the partner
switchover of the the partner site. site.
configuration.
Single Yes Yes for switchover Yes
command for The command for The command for cf takeover
failover or local failover in a switchover is: or
switchover? four-node fabric metrocluster cf forcetakeover -d
MetroCluster switchover
configuration is: or
storage failover metrocluster
takeover switchover
The command for
-forced-on-disaster
switchover is: true
metrocluster
switchover
or
metrocluster
switchover
-forced-on-
disaster true
 Preparing for the MetroCluster installation | 11

Feature or Clustered Data ONTAP Data ONTAP 7-Mode

component
Four node Two node
DS14 disk No No Yes
shelves
supported?
Two FC switch Yes Yes, in fabric Yes
fabrics? MetroCluster
configuration
Stretch No Yes No
configuration
with FC-to-
SAS bridges
(no switch
fabric)?
Stretch No Yes Yes
configuration
with SAS
cables (no
switch fabric)?

Related concepts
Parts of a fabric MetroCluster configuration on page 23

Differences between the clustered Data ONTAP MetroCluster

configurations
The supported MetroCluster configurations have key differences in the required components.

Feature Fabric-attached configurations Stretch configurations

Four-node Two-node Two-node Two-node
bridge-attached direct-attached
Number of Four Two Two Two
controllers
Uses an FC Yes Yes No No
switch storage
fabric
Uses FC-to-SAS Yes Yes Yes No
bridges
Uses direct- No No No Yes
attached SAS
storage
Supports local Yes No No No
HA
Supports No Yes Yes Yes
automatic
switchover
12 | MetroCluster Installation and Configuration Guide

Cluster configuration
In all configurations, each of the two MetroCluster sites is configured as a Data ONTAP cluster. In a
two-node MetroCluster configuration, each node is configured as a single-node cluster.

Considerations for configuring cluster peering

Each MetroCluster site is configured as a peer to its partner site. You should be familiar with the
prerequisites and guidelines for configuring the peering relationships and when deciding whether to
use shared or dedicated ports for those relationships.

Related information
Clustered Data ONTAP 8.3 Data Protection Guide
Clustered Data ONTAP 8.3 Cluster Peering Express Guide

Prerequisites for cluster peering

Before you set up cluster peering, you should confirm that the IPspace, connectivity, port, IP address,
subnet, firewall, and cluster-naming requirements are met.

Connectivity requirements
The subnet used in each cluster for intercluster communication must meet the following
requirements:

The subnet must belong to the broadcast domain that contains the ports used for intercluster
communication.

IP addresses used for intercluster LIFs do not need to be in the same subnet, but having them in
the same subnet is a simpler configuration.

You must have considered whether the subnet will be dedicated to intercluster communication or
shared with data communication.

The intercluster network must be configured so that cluster peers have pair-wise full-mesh
connectivity within the applicable IPspace, which means that each pair of clusters in a cluster peer
relationship has connectivity among all of their intercluster LIFs.
A cluster's intercluster LIFs must use the same IP address version: all IPv4 addresses or all IPv6
addresses. Similarly, all of the intercluster LIFs of the peered clusters must use the same IP
addressing version.

Port requirements
The ports that will be used for intercluster communication must meet the following requirements:

The broadcast domain that is used for intercluster communication must include at least two ports
per node so that intercluster communication can fail over from one port to another.
The ports added to a broadcast domain can be physical network ports, VLANs, or interface
groups (ifgrps).

All of the ports must be cabled.

All of the ports must be in a healthy state.

The MTU settings of the ports must be consistent.

You must have considered whether the ports used for intercluster communication will be shared
with data communication.
 Preparing for the MetroCluster installation | 13

Firewall requirements
Firewalls and the intercluster firewall policy must allow the following:

ICMP service

TCP to the IP addresses of all of the intercluster LIFs over all of the following ports: 10000,
11104, and 11105

HTTPS

The default intercluster firewall policy allows access through the HTTPS protocol and from all
IP addresses (0.0.0.0/0), but the policy can be altered or replaced.

Cluster requirements
Clusters must meet the following requirements:
Each cluster must have a unique name.
You cannot create a cluster peering relationship with any cluster that has the same name or is in a
peer relationship with a cluster of the same name.

The time on the clusters in a cluster peering relationship must be synchronized within 300
seconds (5 minutes).
Cluster peers can be in different time zones.

Considerations when using dedicated ports

When determining whether using a dedicated port for intercluster replication is the correct
intercluster network solution, you should consider configurations and requirements such as LAN
type, available WAN bandwidth, replication interval, change rate, and number of ports.
Consider the following aspects of your network to determine whether using a dedicated port is the
best intercluster network solution:

If the amount of available WAN bandwidth is similar to that of the LAN ports and the replication
interval is such that replication occurs while regular client activity exists, then you should
dedicate Ethernet ports for intercluster replication to avoid contention between replication and the
data protocols.

If the network utilization generated by the data protocols (CIFS, NFS, and iSCSI) is such that the
network utilization is above 50 percent, then you should dedicate ports for replication to allow for
nondegraded performance if a node failover occurs.

When physical 10 GbE ports are used for data and replication, you can create VLAN ports for
replication and dedicate the logical ports for intercluster replication.
The bandwidth of the port is shared between all VLANs and the base port.

Consider the data change rate and replication interval and whether the amount of data that must
be replicated on each interval requires enough bandwidth that it might cause contention with data
protocols if sharing data ports.

Considerations when sharing data ports

When determining whether sharing a data port for intercluster replication is the correct intercluster
network solution, you should consider configurations and requirements such as LAN type, available
WAN bandwidth, replication interval, change rate, and number of ports.
Consider the following aspects of your network to determine whether sharing data ports is the best
intercluster connectivity solution:
14 | MetroCluster Installation and Configuration Guide

For a high-speed network, such as a 10-Gigabit Ethernet (10-GbE) network, a sufficient amount
of local LAN bandwidth might be available to perform replication on the same 10-GbE ports that
are used for data access.
In many cases, the available WAN bandwidth is far less than 10 GbE LAN bandwidth .

All nodes in the cluster might have to replicate data and share the available WAN bandwidth,
making data port sharing more acceptable.

Sharing ports for data and replication eliminates the extra port counts required to dedicate ports
for replication.
The maximum transmission unit (MTU) size of the replication network will be the same size as
that used on the data network.

Consider the data change rate and replication interval and whether the amount of data that must
be replicated on each interval requires enough bandwidth that it might cause contention with data
protocols if sharing data ports.

When data ports for intercluster replication are shared, the intercluster LIFs can be migrated to
any other intercluster-capable port on the same node to control the specific data port that is used
for replication.

Considerations for MetroCluster configurations with native

disk shelves or array LUNs
The MetroCluster configuration supports installations with native (NetApp) disk shelves only, array
LUNs only, or a combination of both.

Related concepts
Planning and installing a MetroCluster configuration with array LUNs on page 154

Related tasks
Cabling a four-node or two-node fabric-attached MetroCluster configuration on page 22

Related information
FlexArray Virtualization Installation Requirements and Reference Guide

Considerations when transitioning from 7-Mode to clustered

Data ONTAP
You must have the new MetroCluster configuration fully configured and operating before you use the
transition tools to move data from a 7-Mode MetroCluster configuration to a Data ONTAP
configuration. If the 7-Mode configuration uses Brocade 6510 switches, the new configuration can
share the existing fabrics to reduce the hardware requirements.
If you have Brocade 6510 switches and plan on sharing the switch fabrics between the 7-Mode fabric
MetroCluster and the MetroCluster running in clustered Data ONTAP, you must use the specific
procedure for configuring the MetroCluster components.
FMC-MCC transition: Configuring the MetroCluster hardware for sharing a 7-Mode Brocade 6510
FC fabric during transition on page 107
 Preparing for the MetroCluster installation | 15

Configuration of new MetroCluster systems

New MetroCluster components are preconfigured and MetroCluster settings are enabled in the
software. In most cases, you do not need to perform the detailed procedures provided in this guide.

Hardware racking and cabling

Depending on the configuration you ordered, you might need to rack the systems and complete the
cabling.
Configuring the MetroCluster hardware components in systems with native disk shelves on page 22

FC switch and FC-to-SAS bridge configuration

For configurations using FC-to-SAS bridges, the bridges received with the new MetroCluster
configuration are preconfigured and do not require additional configuration unless you want to
change the names and IP addresses.
For configurations using FC switches, in most cases, FC switch fabrics received with the new
MetroCluster configuration are preconfigured for two Inter-Switch Links (ISLs). If you are using
three or four ISLs, you must manually configure the switches.
Configuring the FC switches on page 49

MetroCluster configuration in Data ONTAP

Nodes and clusters received with the new MetroCluster configuration are preconfigured and the
MetroCluster configuration is enabled.
Configuring the FC switches on page 49

Preconfigured component passwords

Some MetroCluster components are preconfigured with usernames and passwords. You need to be
aware of these settings as you perform your site-specific configuration.

Component Username Password

Data ONTAP login admin netapp!123
Service Processor (SP) login admin netapp!123
Intercluster pass phrase None required netapp!123
ATTO FC-to-SAS bridge None required None required
Brocade switches Admin password
NetApp cluster interconnect None required None required
switches (CN1601, CN1610)

Hardware setup checklist

You need to know which hardware setup steps were completed at the factory and which steps you
need to complete at each MetroCluster site.

Step Completed Completed by you

at factory
Mount components in one or more cabinets. Yes No
16 | MetroCluster Installation and Configuration Guide

Step Completed Completed by you

at factory
Position cabinets in the desired location. No Yes
Position them in the original order
to ensure that the supplied cables
are long enough.
Connect multiple cabinets to each other, if No Yes
applicable. Use the cabinet interconnect kit if it
is included in the order. The kit box
is labeled.
Secure the cabinets to the floor, if No Yes
applicable. Use the universal bolt-down kit if it
is included in the order. The kit box
is labeled.
Cable the components within the cabinet. Yes No
Cables 5
meters and
longer are
removed for
shipping and
placed in the
accessories
box.
Connect the cables between cabinets, if No Yes
applicable. Cables are in the accessories box.
Connect management cables to the No Yes
customer's network. Connect them directly or through
the CN1601 management switches,
if present.
Attention: To avoid address
conflicts, do not connect
management ports to the
customer's network until after you
change the default IP addresses to
the customer's values.

Connect console ports to the customer's No Yes

terminal server, if applicable.
Connect the customer's data cables to the No Yes
cluster.
Connect the long-distance ISLs between the No Yes
MetroCluster sites, if applicable. Connecting the ISLs between the
MetroCluster sites on page 37
 Preparing for the MetroCluster installation | 17

Step Completed Completed by you

at factory
Connect the cabinets to power and power on No Yes
the components. Power them on in the following
order:

1. PDUs

2. Disk shelves and FC-to-SAS

bridges, if applicable

3. FC switches, if applicable

4. Nodes

Assign IP addresses to the management No Yes, for switched clusters only

ports of the cluster switches and to the Connect to the serial console port of
management ports of the management each switch and log in with user
switches if present. name admin with no password.
Suggested addresses are
10.10.10.81, 10.10.10.82,
10.10.10.83, and 10.10.10.84.
Verify cabling by running the Config No Yes
Advisor tool. Verifying the MetroCluster
configuration on page 151

Software setup checklist

You need to know which software setup steps were completed at the factory and which steps you
need to complete at each MetroCluster site.
This guide includes all the required steps, which you can complete after reviewing the information in
this checklist.

Step Completed at factory Completed by you using

procedures in this guide
Install the clustered Data Yes No
ONTAP software.
18 | MetroCluster Installation and Configuration Guide

Step Completed at factory Completed by you using

procedures in this guide
Create the cluster on the first Yes No
node at the first MetroCluster
site:

Name the cluster.

Set the admin password.

Set up the private cluster

interconnect.
Install all purchased license
keys.

Create the cluster

management interface.

Create the node

management interface.

Configure the FC switches,

if applicable.

In a four-node MetroCluster Yes No

configuration, join the
remaining nodes to the cluster.
In a four-node MetroCluster Yes No
configuration, enable storage
failover on one node of each
HA pair and configure cluster
high availability.
Enable the switchless-cluster Yes No
option on a two-node
switchless cluster.
Repeat the steps to configure Yes No
the second MetroCluster site.
Configure the clusters for Yes No
peering.
Enable the MetroCluster Yes No
configuration.
Configure user credentials and Yes, if ordered No
management IP addresses on User IDs are admin with no
the management and cluster password.
switches.
Thoroughly test the Yes No, although you must perform
MetroCluster configuration. verification steps at your site as
described below.
Complete the cluster setup No Yes
worksheet.
 Preparing for the MetroCluster installation | 19

Step Completed at factory Completed by you using

procedures in this guide
Change the password for the No Yes
admin account to the
customer's value.
Configure each node with the No Yes
customer's values.
Discover the clusters in No Yes
OnCommand System
Manager.
Configure an NTP server for No Yes
each cluster.
Verify the cluster peering. No Yes
Verify the health of the cluster No Yes
and that the cluster is in
quorum.
Verify basic operation of the No Yes.
MetroCluster sites.
Check the MetroCluster No Yes
configuration.
In a four-node MetroCluster No Yes
configuration, test storage
failover.
If the configuration includes No Yes
FC switches or FC-to-SAS
bridges, add the MetroCluster
switches and bridges for health
monitoring.
Test switchover, healing, and No Yes
switchback.
Set the destination for No Yes
configuration backup files.
Optional: Change the cluster No Yes
name if desired, for example,
to better distinguish the
clusters.
Optional: Change the node No Yes
name, if desired.
Configure AutoSupport. No Yes
20

Choosing the correct installation procedure for

your configuration
You need to choose the correct installation procedure based on whether you are using FlexArray
LUNs, the number of nodes in the MetroCluster configuration, or if you are sharing an existing FC
switch fabric used by a 7-Mode fabric MetroCluster.

For this installation type... Use these procedures...

Four-node or two-node fabric
1. Cabling a four-node or two-node fabric-attached
configuration
MetroCluster configuration on page 22
2. Configuring the MetroCluster software in Data
ONTAP (native disk shelves only) on page 116
 Choosing the correct installation procedure for your configuration | 21

For this installation type... Use these procedures...

Two-node stretch configuration with
1. Cabling a two-node bridge-attached stretch
FC-to SAS bridges
MetroCluster configuration on page 94
2. Configuring the MetroCluster software in Data
ONTAP (native disk shelves only) on page 116

Two-node stretch configuration with

1. Cabling a two-node SAS-attached stretch
direct-attached SAS cabling
MetroCluster configuration on page 87
2. Configuring the MetroCluster software in Data
ONTAP (native disk shelves only) on page 116

Installation with array LUNs Planning and installing a MetroCluster configuration

with array LUNs on page 154
Installation when sharing with an
1. Configuring the MetroCluster hardware for sharing
existing FC switch fabric
a 7-Mode Brocade 6510 FC fabric during transition
This is supported only as a temporary on page 107
configuration with a 7-Mode fabric
MetroCluster using Brocade 6510 2. Configuring the MetroCluster software in Data
switches. ONTAP (native disk shelves only) on page 116
22

Cabling a four-node or two-node fabric-attached

MetroCluster configuration
The MetroCluster components must be physically installed, cabled, and configured at both
geographic sites. The steps are slightly different for a system with native disk shelves as opposed to a
system with array LUNs.

About this task

Steps
1. Parts of a fabric MetroCluster configuration on page 23
2. Required MetroCluster components and naming guidelines for fabric configurations on page 29
3. Worksheet for FC switches and FC-to-SAS bridges on page 31
4. Installing and cabling MetroCluster components on page 33
5. Installing FC-to-SAS bridges and SAS disk shelves on page 43
6. Configuring the FC switches on page 49
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 23

Parts of a fabric MetroCluster configuration

The MetroCluster configuration consists of two separate clusters at physically separated sites and the
networking infrastructure between the sites.

Key hardware elements

The MetroCluster configuration includes the following key hardware elements:
Storage controllers
The storage controllers are not connected directly to the storage but connect to two redundant FC
switch fabrics.
Each storage controller is configured as a DR partner to a storage controller on the partner site.
When the MetroCluster is enabled, the system will automatically pair the two nodes with lowest
system IDs in each of the two clusters as DR partners.
Each storage controller is configured as a DR partner to a storage controller on the partner site.
When the MetroCluster is enabled, the system will automatically pair the two nodes with lowest
system IDs in each of the two clusters as DR partners. In a four-node MetroCluster, there are two
DR partner pairs. The second DR pair is created from the two nodes with higher system IDs.

FC-to-SAS bridges
The FC-to-SAS bridges connect the SAS storage stacks to the FC switches, providing bridging
between the two protocols.

FC switches
The FC switches provide the long haul backbone ISL between the two sites. The switches provide
the two storage fabrics that allow data mirroring to the remote storage pools.

Cluster peering network

The cluster peering network provides connectivity for mirroring of the Storage Virtual Machine
(SVM) configuration. The configuration of all SVMs on one cluster is mirrored to the partner
cluster.

Four-node fabric MetroCluster configuration

The following illustration shows a simplified view of a four-node fabric MetroCluster configuration.
For some connections, a single line represents multiple, redundant connections between the
components. Data and management network connections are not shown.
24 | MetroCluster Installation and Configuration Guide

Cluster peering network

FC_switch_A_1 FC_switch_B_1
controller_A_1 controller_B_1
FC Long-haul ISLs
FC
cluster interconnect

cluster interconnect
HA interconnect FC_bridge_A_1 FC_bridge_B_1

HA interconnect
SAS stack SAS stack

FC_bridge_A_2 FC_bridge_B_2

FC FC
controller_A_2 Long-haul ISLs controller_B_2
FC_switch_A_2 FC_switch_B_2

cluster_A cluster_B

The configuration consists of two clusters, one at each geographically separated site.

cluster_A is located at one MetroCluster site.

cluster_B is located at the second MetroCluster site.

Each site has one stack of SAS storage.

Additional storage stacks are supported, but only one is shown at each site.

The HA pairs are configured as switchless clusters, without cluster interconnect switches.
A switched configuration is supported but not shown.

The following illustration shows a more detailed view of the connectivity in a single MetroCluster
cluster (both clusters have the same configuration):
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 25

The configuration includes the following connections:

FC connections from each controller's HBAs and FC-VI adapters to each of the FC switches.

An FC connection from each FC-to-SAS bridge to an FC switch.

SAS connections between each SAS shelf and from the top and bottom of each stack to an FC-to-
SAS bridge.

An HA interconnect between each controller in the local HA pair.

If the controllers support a single-chassis HA pair, the HA interconnect is internal, occurring
through the backplane, meaning an external interconnect is not required.

Ethernet connections from the controllers to the customer-provided network used for cluster
peering.
SVM configuration is replicated over the cluster peering network.

A cluster interconnect between each controller in the local HA pair.

If the controllers are configured as a switched cluster, each controller would connect to two
cluster interconnect switches.

Two-node fabric MetroCluster configuration

The following illustration shows a simplified view of a two-node fabric MetroCluster configuration.
For some connections, a single line represents multiple, redundant connections between the
components. Data and management network connections are not shown.
26 | MetroCluster Installation and Configuration Guide

Cluster peering network

FC_switch_A_1 FC_switch_B_1
controller_A_1 controller_B_1
FC Long-haul ISLs
FC

FC_bridge_A_1 FC_bridge_B_1

SAS stack SAS stack

FC_bridge_A_2 FC_bridge_B_2

FC FC
Long-haul ISLs
FC_switch_A_2 FC_switch_B_2

cluster_A cluster_B

The configuration consists of two clusters, one at each geographically separated site.

cluster_A is located at one MetroCluster site.

cluster_B is located at the second MetroCluster site.

Each site has one stack of SAS storage.

Additional storage stacks are supported, but only one is shown at each site.

Note: In the two-node configuration, the nodes are not configured as an HA pair.

The following illustration shows a more detailed view of the connectivity in a single MetroCluster
cluster (both clusters have the same configuration):

Cluster peering network

to partner
site

FC_switch_A_1
Long-haul
FC ISL

FC_bridge_A_1

controller_A_1
SAS-attached shelf

SAS-attached shelf

SAS-attached shelf

FC_bridge_A_2
Ethernet (controller_A_1)
Long-haul
FC ISL
Fibre Channel (controller_A_1) FC_switch_A_2

Fibre Channel (bridge to switch)

The configuration includes the following connections:

 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 27

FC connections between the FC-VI adapter on each controller module.

FC connections from each controller module's HBAs to FC-to-SAS bridge for each SAS shelf
stack.

SAS connections between each SAS shelf and from the top and bottom of each stack to an FC-to-
SAS bridge.

Ethernet connections from the controllers to the customer-provided network used for cluster
peering.
SVM configuration is replicated over the cluster peering network.

Local HA pairs
In a four-node MetroCluster configuration, each site consists of two storage controllers configured as
an HA pair. This allows local redundancy so that if one storage controller fails, its local HA partner
can take over. Such failures can be handled without a MetroCluster switchover operation.
Local HA failover and giveback operations are performed with the storage failover commands,
in the same manner as a non-MetroCluster configuration.

Cluster peering network

FC_switch_A_1 FC_switch_B_1
controller_A_1 controller_B_1
FC Long-haul ISL
FC
(one to four)
cluster interconnect

cluster interconnect
FC_bridge_A_1 FC_bridge_B_1
HA interconnect

HA interconnect
SAS stack SAS stack

FC_bridge_A_2 FC_bridge_B_2

FC FC
controller_A_2 Long-haul ISL controller_B_2
FC_switch_A_2 FC_switch_B_2
(one to four)

cluster_A cluster_B

HA pair, Site A HA pair, Site B

Related information
Clustered Data ONTAP 8.3 High-Availability Configuration Guide

Redundant FC-to-SAS bridges

FC-to-SAS bridges provide protocol bridging between SAS attached disks and the FC switch fabric.
Each disk shelf stack requires two FC-to-SAS bridges.
28 | MetroCluster Installation and Configuration Guide

Cluster peering network

FC_switch_A_1 FC_switch_B_1
controller_A_1 controller_B_1
FC Long-haul ISL
FC
(one or two)
cluster interconnect

cluster interconnect
FC_bridge_A_1 FC_bridge_B_1
HA interconnect

HA interconnect
SAS stack SAS stack

FC_bridge_A_2 FC_bridge_B_2

FC FC
controller_A_2 Long-haul ISL controller_B_2
FC_switch_A_2 FC_switch_B_2
(one or two)

cluster_A cluster_B

Shelf stack, Site A Shelf stack, Site B

Redundant FC switch fabrics

Each switch fabric includes interswitch links (ISLs) that connect the sites. Data is replicated from
site-to-site over the ISL.

FC fabric 1
Cluster peering network

FC_switch_A_1 FC_switch_B_1
controller_A_1 controller_B_1
FC Long-haul ISL
FC
(one or two)
cluster interconnect

cluster interconnect

FC_bridge_A_1 FC_bridge_B_1
HA interconnect

HA interconnect

SAS stack SAS stack

FC_bridge_A_2 FC_bridge_B_2

FC FC
controller_A_2 Long-haul ISL controller_B_2
FC_switch_A_2 FC_switch_B_2
(one or two)

cluster_A cluster_B
FC fabric 2

The cluster peering network

The two clusters in the MetroCluster configuration are peered through a customer-provided cluster
peering network. Cluster peering supports the synchronous mirroring of Storage Virtual Machines
(SVMs, formerly known as Vservers) between the sites.
Intercluster LIFs must be configured on each node in the MetroCluster configuration, and the clusters
must be configured for peering. The ports with the intercluster LIFs are connected to the customer-
provided cluster peering network. Replication of SVM configuration is carried out over this network
through the Configuration Replication Service.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 29

Cluster peering network

FC_switch_A_1 FC_switch_B_1
controller_A_1 controller_B_1
FC Long-haul ISL
FC
(one or two)
cluster interconnect

cluster interconnect
FC_bridge_A_1 FC_bridge_B_1
HA interconnect

HA interconnect
SAS stack SAS stack

FC_bridge_A_2 FC_bridge_B_2

FC FC
controller_A_2 Long-haul ISL controller_B_2
FC_switch_A_2 FC_switch_B_2
(one or two)

cluster_A cluster_B

Related concepts
Considerations for configuring cluster peering on page 12

Related tasks
Cabling the cluster peering connections on page 41
Peering the clusters on page 135

Related information
Clustered Data ONTAP 8.3 Cluster Peering Express Guide

Required MetroCluster components and naming guidelines

for fabric configurations
The MetroCluster configuration requires a variety of hardware components. For convenience and
clarity, standard names for components are used throughout these procedures. Also, one site is
referred to as Site A and the other site is referred to as Site B. Such terms are relative.

Supported software and hardware

The hardware and software must be supported for the MetroCluster configuration.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray). You
use the Component Explorer to select the components and Data ONTAP version to refine your
search. You can click Show Results to display the list of supported configurations that match the
criteria.
NetApp Hardware Universe
When using All Flash Optimized systems, all controller modules in the MetroCluster configuration
must be configured as All Flash Optimized systems.
30 | MetroCluster Installation and Configuration Guide

Required components
Because of the hardware redundancy in the MetroCluster configuration, there are two of each
component at each site. The sites are arbitrarily assigned the letters A and B and the individual
components are arbitrarily assigned the numbers 1 and 2.
The MetroCluster configuration also includes SAS storage shelves that connect to the FC-to-SAS
bridges.

Component Present in Present in Example names

two-node four-node
Site A Site B
configuratio configuratio
n? n?
Two Data ONTAP clusters, one Yes Yes cluster_A cluster_B
at each MetroCluster site
Naming must be unique within
the MetroCluster configuration.
Four FC switches (supported Yes Yes FC_switch_A_ FC_switch_
Brocade or Cisco models) 1 B_1
The four switches form two FC_switch_A_ FC_switch_
switch storage fabrics that 2 B_2
provide the ISL between each of
the clusters in the MetroCluster
configuration.
Up to four ISLs per fabric are
supported.
Naming must be unique within
the MetroCluster configuration.
Storage controllers One The two controller_A_1 controller_B
Naming must be unique within controller at controllers at _1
the MetroCluster configuration. each site. each site
controller_A_2 controller_B
Each form an HA
All controller modules must be _2
controller is pair. Each
the same platform model and
a DR partner controller
running the same version of Data
for the other has a DR
ONTAP.
controller. partner at the
other site.
Four cluster interconnect No Yes clus_sw_A_1 clust_sw_B_
switches (if not using two-node 1
switchless clusters)
clus_sw_A_2 clust_sw_B_
These switches provide cluster 2
communication among the
storage controllers in each
cluster. The switches are not
required if the storage controllers
at each site are configured as a
two-node switchless cluster.
Naming must be unique within
the MetroCluster configuration.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 31

Component Present in Present in Example names

two-node four-node
Site A Site B
configuratio configuratio
n? n?
Two FC-to-SAS bridges (ATTO Yes Yes bridge_A_1_po bridge_B_1_
6500N FibreBridge bridges) per rt-number port-
storage stack number
These connect the SAS disk bridge_A_2_po bridge_B_2_
shelves to the FC switch fabric. rt-number port-
Naming must be unique within number
the MetroCluster configuration.
The suggested names used as
examples in this guide identify
the controller that the bridge
connects to and the port.
At least eight SAS disk shelves Yes Yes shelf_A_1_1 shelf_B_1_1
(recommended)
shelf_A_2_1 shelf_B_2_1
Four shelves are recommended at
each site to allow disk ownership shelf_B_1_2 shelf_A_1_2
on a per-shelf basis. A minimum shelf_B_2_2 shelf_A_2_2
of two shelves at each site is
supported.
Note: FlexArray systems
support array LUNs and have
different storage requirements.
Requirements for a
MetroCluster configuration
with array LUNs on page 155

Worksheet for FC switches and FC-to-SAS bridges

Before beginning to configure the MetroCluster sites, you should gather required configuration
information.

Site A, FC switch one (FC_switch_A_1)

Switch Your value

configuration
parameter
FC_switch_A_1 IP
address
FC_switch_A_1
Username
FC_switch_A_1
Password
32 | MetroCluster Installation and Configuration Guide

Site A, FC switch two (FC_switch_A_2)

Switch Your value

configuration
parameter
FC_switch_A_2 IP
address
FC_switch_A_2
Username
FC_switch_A_2
Password

Site A, FC-to-SAS bridge 1 (FC_bridge_A_1_port-number)

Each SAS stack requires two FC-to-SAS bridges. One bridge connects to FC_switch_A_1_port-
number and the second connects to FC_switch_A_2_port-number.

Site A Your value

Bridge_A_1_port-number IP address
Bridge_A_1_port-number Username
Bridge_A_1_port-number Password

Site A, FC-to-SAS bridge 2 (FC_bridge_A_2_port-number)

Each SAS stack requires two FC-to-SAS bridges. One bridge connects to FC_switch_A_1_port-
number and the second connects to FC_switch_A_2_port-number.

Site A Your value

Bridge_A_2_port-number IP address
Bridge_A_2_port-number Username
Bridge_A_2_port-number Password

Site B, FC switch one (FC_switch_B_1)

Site B Your value

FC_switch_B_1 IP address
FC_switch_B_1 Username
FC_switch_B_1 Password

Site B, FC switch two (FC_switch_B_2)

Site B Your value

FC_switch_B_2 IP address
FC_switch_B_2 Username
FC_switch_B_2 Password
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 33

Site B, FC-to-SAS bridge 1 (FC_bridge_B_1_port-number)

Each SAS stack requires two FC-to-SAS bridges. One bridge connects to FC_switch_B_1_port-
number and the second connects to FC_switch_B_2_port-number.

Site B Your value

Bridge_B_1_port-number IP address
Bridge_B_1_port-number Username
Bridge_B_1_port-number Password

Site B, FC-to-SAS bridge 2 (FC_bridge_B_2_port-number)

Each SAS stack requires two FC-to-SAS bridges. One bridge connects to FC_switch_B_1_port-
number and the second connects to FC_switch_B_2_port-number.

Site B Your value

Bridge_B_2_port-number IP address
Bridge_B_2_port-number Username
Bridge_B_2_port-number Password

Installing and cabling MetroCluster components

The storage controllers must be cabled to the FC switches and the ISLs must be cabled to link the
MetroCluster sites. The storage controllers must also be cabled to the data and management network.

Steps
1. Racking the hardware components on page 33
2. Cabling the HBA adapters to the FC switches on page 34
3. Cabling the ISLs between MetroCluster sites on page 37
4. Recommended port assignments for FC switches on page 38
5. Cabling the cluster interconnect in four-node configurations on page 41
6. Cabling the cluster peering connections on page 41
7. Cabling the HA interconnect, if necessary on page 42
8. Cabling the management and data connections on page 43

Racking the hardware components

If you have not received the equipment already installed in cabinets, you must rack the components.

About this task

This task must be performed on both MetroCluster sites.

Steps

1. Plan out the positioning of the MetroCluster components.

The rack space depends on the platform model of the storage controllers, the switch types, and the
number of disk shelf stacks in your configuration.

2. Properly ground yourself.

34 | MetroCluster Installation and Configuration Guide

3. Install the storage controllers in the rack or cabinet.

Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems
4. Install the FC switches in the rack or cabinet.

5. Install the disk shelves, power them on, and set the shelf IDs.
SAS Disk Shelves Installation and Service Guide for DS4243, DS2246, DS4486, and DS4246
You must power-cycle each disk shelf.

Shelf IDs must be unique for each SAS disk shelf within the entire MetroCluster configuration
(including both sites).

6. Install each FC-to-SAS bridge:

a. Secure the L brackets on the front of the bridge to the front of the rack (flush-mount) with
the four screws.
The openings in the bridge L brackets are compliant with rack standard ETA-310-X for 19-
inch (482.6 mm) racks.
For more information and an illustration of the installation, see the ATTO FibreBridge 6500N
Installation and Operation Manual.
b. Connect each bridge to a power source that provides a proper ground.

c. Power on each bridge.

Note: For maximum resiliency, ensure that bridges attached to the same stack of disk
shelves are connected to different power sources.

The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge
has completed its power-on self test sequence.

Cabling the HBA adapters to the FC switches

The HBA adapters must be cabled to the site FC switches on each controller module in the
MetroCluster configuration.

About this task

This task must be performed on both MetroCluster sites.

Step

1. Cable the HBA ports.

 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 35

Are you And Then use this cabling...

using dual your
or single FC switch
connections vendor
to each is...
switch?
Dual Brocade
connections. Controller 1 HBA Controller 2 HBA

port a port b port c port d port a port b port c port d

switch_A_1

0 1 2 3 4 5 6 7 8 9 10 11

switch_A_2

0 1 2 3 4 5 6 7 8 9 10 11

Cisco
Controller 1 HBA Controller 2 HBA

port a port b port c port d port a port b port c port d

switch_A_1

1 2 3 4 5 6 7 8 9 10 11 12

switch_A_2

1 2 3 4 5 6 7 8 9 10 11 12
36 | MetroCluster Installation and Configuration Guide

Are you And Then use this cabling...

using dual your
or single FC switch
connections vendor
to each is...
switch?
Controller 1 HBA Controller 2 HBA

port a port b port c port d port a port b port c port d

switch_A_1

0 1 2 3 4 5 6 7 8 9 10 11

switch_A_2

0 1 2 3 4 5 6 7 8 9 10 11

Single Brocade
connections.
Controller 1 HBA Controller 2 HBA
This is
port a port b port c port d port a port b port c port d
supported on
8020
systems
only.
switch_A_1

0 1 2 3 4 5 6 7 8 9 10 11

switch_A_2

0 1 2 3 4 5 6 7 8 9 10 11

Cisco

Note: The Brocade and Cisco switches use different port numbering:

On Brocade switches, the first port is numbered 0.

On Cisco switches, the first port is numbered 1.

The following tables show both Brocade and Cisco port numbering:
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 37

Cabling to FC_switch_x_1
Single or dual FC connection to Connect this site To this port on
each switch? component and port... FC_switch_x_1...
Brocade Cisco
Dual controller_x_1 HBA 1 2
port a
controller_x_1 HBA 2 3
port c
controller_x_2 HBA 4 5
port a
controller_x_2 HBA 5 6
port c
Single controller_x_1 HBA 1 2
port c
controller_x_2 HBA 4 5
port c

Cabling to FC_switch_x_2
Single or dual FC connection to Connect this site To this port on
each switch? component and port... FC_switch_x_2...
Brocade Cisco
Dual controller_x_1 HBA 1 2
port b
controller_x_1 HBA 2 3
port d
controller_x_2 HBA 4 5
port b
controller_x_2 HBA 5 6
port d
Single controller_x_1 HBA 1 2
port d
controller_x_2 HBA 4 5
port d

Related concepts
Port assignments for FC switches in a four-node configuration on page 38

Cabling the ISLs between MetroCluster sites

You must connect the FC switches at each site through the fiber-optic Inter-Switch Links (ISLs) to
form the switch fabrics that connect the MetroCluster components.

About this task

This task includes steps performed at each MetroCluster site.
Up to four ISLs per FC switch fabric are supported.
38 | MetroCluster Installation and Configuration Guide

Step

1. Connect the FC switches at each site to the ISL or ISLs.

This must be done for both switch fabrics.
Note: The Brocade and Cisco switches use different port numbering:

On Brocade switches, the first port is numbered 0.

On Cisco switches, the first port is numbered 1.

The port numbering is shown in the following table:

ISL Site A Site B

To this port on FC_switch_A_x... To this port on FC_switch_B_x...
Brocade Brocade 6510 Cisco Brocade Brocade Cisco
6505 9148 or 6505 6510 9148 or
Cisco Cisco
9148S 9148S
First ISL 8 20 36 8 20 36
Second ISL 9 21 40 9 21 40
Third ISL 10 22 44 10 22 44
Four ISL 11 23 48 11 23 48

Related concepts
Port assignments for FC switches in a four-node configuration on page 38
Port assignments for FC switches in a two-node configuration on page 40

Recommended port assignments for FC switches

The recommended ports are different, depending on whether you are in a four-node or two-node
configuration.

Port assignments for FC switches in a four-node configuration

You need to verify that you are using the specified port assignments when you cable the FC switches.
The Brocade and Cisco switches use different port numbering:

On Brocade switches, the first port is numbered 0.

On Cisco switches, the first port is numbered 1.

This difference is reflected in the following table.

Note: The cabling is the same for each FC switch in the switch fabric.

FC_switch_x_1 FC_switch_x_2
Component and port Brocad Brocade Cisco Brocade Brocade Cisco
e 6505 6510 9148 or 6505 6510 9148 or
Cisco Cisco
9148S 9148S
controller_x_1 FC-VI 0 0 1 - - -
port a
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 39

FC_switch_x_1 FC_switch_x_2
Component and port Brocad Brocade Cisco Brocade Brocade Cisco
e 6505 6510 9148 or 6505 6510 9148 or
Cisco Cisco
9148S 9148S
controller_x_1 FC-VI - - - 0 0 1
port b
controller_x_1 HBA 1 1 2 - - -
port a
controller_x_1 HBA - - - 1 1 2
port b
controller_x_1 HBA 2 2 3 - - -
port c
controller_x_1 HBA - - - 2 2 3
port d
controller_x_2 FC-VI 3 3 4 - - -
port a
controller_x_2 FC-VI - - - 3 3 4
port b
controller_x_2 HBA 4 4 5 - - -
port a
controller_x_2 HBA - - - 4 4 5
port b
controller_x_2 HBA 5 5 6 - - -
port c
controller_x_2 HBA - - - 5 5 6
port d
bridge_x_1_port- 6 6 7 6 6 7
number port 1

bridge_x_1_port- 7 7 8 7 7 8
number port 1

bridge_x_1_port- 12 8 9 12 8 9
number port 1

bridge_x_1_port- 13 9 10 13 9 10
number port 1

ISL port 1 8 20 36 8 20 36
ISL port 2 9 21 40 9 21 40
ISL port 3 10 22 44 10 22 44
ISL port 4 11 23 48 11 23 48

Related tasks
Cabling the HBA adapters to the FC switches on page 34
Cabling the ISLs between MetroCluster sites on page 37
40 | MetroCluster Installation and Configuration Guide

Port assignments for FC switches in a two-node configuration

You need to verify that you are using the specified port assignments when you cable the FC switches.
The Brocade and Cisco switches use different port numbering:

On Brocade switches, the first port is numbered 0.

On Cisco switches, the first port is numbered 1.

This difference is reflected in the following table.

Note: The cabling is the same for each FC switch in the switch fabric.

FC_switch_x_1 FC_switch_x_2
Component and port Brocad Brocade Cisco Brocade Brocade Cisco
e 6505 6510 9148 or 6505 6510 9148 or
9148S 9148S
controller_x_1 FC-VI 0 0 1 - - -
port a
controller_x_1 FC-VI - - - 0 0 1
port b
controller_x_1 HBA 1 1 2 - - -
port a
controller_x_1 HBA - - - 1 1 2
port b
controller_x_1 HBA 2 2 3 - - -
port c
controller_x_1 HBA - - - 2 2 3
port d
bridge_x_1_port- 6 6 7 6 6 7
number port 1

bridge_x_1_port- 7 7 8 7 7 8
number port 1

bridge_x_1_port- 12 8 9 12 8 9
number port 1

bridge_x_1_port- 13 9 10 13 9 10
number port 1

ISL port 1 8 20 36 8 20 36
ISL port 2 9 21 40 9 21 40
ISL port 3 10 22 44 10 22 44
ISL port 4 11 23 48 11 23 48
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 41

Cabling the cluster interconnect in four-node configurations

In a four-node MetroCluster configuration, you must cable the cluster interconnect between the local
controllers at each site. If the local controllers are not configured as two-node switchless clusters,
two-cluster switches are required.

About this task

This task is not required on two-node MetroCluster configurations.
This task must be performed at both MetroCluster sites.

Step

1. Cable the cluster interconnect from one controller to the other, or, if cluster interconnect switches
are used, from each controller to the switches.

If the storage Do this if using cluster Do this if using a switchless

controller is a... interconnect switches... cluster...
80xx Connect ports e0a through e0d on Connect ports e0a through e0d on
each controller to a local cluster each controller to the same ports on
interconnect switch. its HA partner.
62xx Connect ports e0c and e0e on each Connect ports e0c and e0e on each
controller to a local cluster controller to the same ports on its HA
interconnect switch. partner.
32xx Connect ports e1a and e2a on each Connect ports e1a and e2a on each
controller to a local cluster controller to the same ports on its HA
interconnect switch. partner.

Related information
Clustered Data ONTAP 8.3 Network Management Guide

Cabling the cluster peering connections

You must ensure that the controller ports used for cluster peering have connectivity with the cluster
on the partner site.

About this task

This task must be performed on each controller in the MetroCluster configuration.
At least two ports on each controller should be used for cluster peering.
The recommended minimum bandwidth for the ports and network connectivity is 1 GbE.

Step

1. Identify and cable at least two ports for cluster peering and ensure they have network connectivity
with the partner cluster.
Cluster peering can be done on dedicated ports or on data ports. Using dedicated ports ensures
higher throughput for the cluster peering traffic.

Related concepts
Considerations for configuring cluster peering on page 12
42 | MetroCluster Installation and Configuration Guide

Related information
Clustered Data ONTAP 8.3 Data Protection Guide
Clustered Data ONTAP 8.3 Cluster Peering Express Guide

Cabling the HA interconnect, if necessary

If you have a four-node MetroCluster configuration and the storage controllers in the HA pair are in
separate chassis, you must cable the HA interconnect between the controllers.

About this task

This task does not apply to two-node MetroCluster configurations.

This task must be performed at both MetroCluster sites.

The HA interconnect must be cabled only if the storage controllers in the HA pair are in separate
chassis.
Some storage controller models support two controllers in a single chassis, in which case they use
an internal HA interconnect.

Steps

1. Cable the HA interconnect if the storage controller's HA partner is in a separate chassis.

If the storage controller is Do this...

a...
80xx with I/O Expansion
a. Connect port ib0a on the first controller in the HA pair to port ib0a on
Module
the other controller.

b. Connect port ib0b on the first controller in the HA pair to port ib0b
on the other controller.

62xx
a. Connect port 2a (top port of NVRAM8 card in vertical slot 2) on the
first controller in the HA pair to port 2a on the other controller.

b. Connect port 2b (bottom port of NVRAM8 card in vertical slot 2) on

the first controller in the HA pair to port 2b on the other controller.

32xx
a. Connect port c0a on the first controller in the HA pair to port c0a on
the other controller.

b. Connect port c0b on the first controller in the HA pair to port c0b on
the other controller.

2. Repeat this task at the MetroCluster partner site.

Related information
Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 43

Cabling the management and data connections

You must cable the management and data ports on each storage controller to the site networks.

About this task

This task must be repeated for each controller at both MetroCluster sites.
You can connect the controller and cluster switch management ports to existing switches in your
network or to new dedicated network switches such as NetApp CN1601 cluster management
switches.

Step

1. Cable the controller's management and data ports to the management and data networks at the
local site.
Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems

Installing FC-to-SAS bridges and SAS disk shelves

You install and cable ATTO FibreBridge FC-to-SAS bridges and SAS disk shelves as part of a new
MetroCluster installation.

About this task

For systems received from the factory, the FC-to-SAS bridges are preconfigured and do not require
additional configuration
This procedure is written with the assumption that you are using the recommended bridge
management interfaces: the ATTO ExpressNAV GUI and ATTO QuickNAV utility.
You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge
firmware. You use the ATTO QuickNAV utility to configure the bridge Ethernet management 1 port.
You can use other management interfaces instead, if needed, such as a serial port or Telnet to
configure and manage a bridge and to configure the Ethernet management 1 port, and FTP to update
the bridge firmware.
This procedure uses the following workflow:
44 | MetroCluster Installation and Configuration Guide

Steps
1. Preparing for the installation on page 44
2. Installing the FC-to-SAS bridge and SAS shelves on page 45

Related concepts
Example of a four-node MetroCluster configuration with disks and array LUNs on page 194

Preparing for the installation

When you are preparing to install the FC-to-SAS bridges as part of your new MetroCluster system,
you must ensure that your system meets certain requirements, including meeting setup and
configuration requirements for the bridges. Other requirements include downloading the necessary
documents, the ATTO QuickNAV utility, and the bridge firmware.

Before you begin

Your system must already be installed in a rack if it was not shipped in a system cabinet.

Your configuration must be using supported hardware models and software versions.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

Each FC switch must have one FC port available for one bridge to connect to it.

The computer you are using to set up the bridges must be running an ATTO-supported web
browser to use the ATTO ExpressNAV GUI.
The ATTO-supported web browsers are Internet Explorer 8 and 9, and Mozilla Firefox 3.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 45

The ATTO Product Release Notes have an up-to-date list of supported web browsers. You can
access this document from the ATTO web site as described in the following steps.

Steps

1. Download the following documents:

SAS Disk Shelves Installation and Service Guide for DS4243, DS2246, DS4486, and DS4246
2. Download content from the ATTO web site and from the NetApp web site:

a. From NetApp Support, navigate to the ATTO FibreBridge Description page by clicking
Software, scrolling to Protocol Bridge and choosing ATTO FibreBridge from the drop-
down menu

b. Click Go!, and then clicking View & Download.

c. Access the ATTO web site using the link provided and download the following:

ATTO FibreBridge 6500N Installation and Operation Manual

ATTO QuickNAV utility (to the computer you are using for setup)

d. Navigate to the ATTO FibreBridge 6500N Firmware Download page by clicking Continue
at the end of the ATTO FibreBridge Description page.

Download the bridge firmware file using Steps 1 through 3 of that procedure.
You update the firmware on each bridge later in this procedure.

Make a copy of the ATTO FibreBridge 6500N Firmware Download page and release notes
for reference when you are instructed to update the firmware on each bridge.

3. Gather the hardware and information needed to use the recommended bridge management
interfaces, the ATTO ExpressNAV GUI, and the ATTO QuickNAV utility:

a. Acquire a standard Ethernet cable.

The cable connects from the bridge Ethernet management 1 port to your network.

b. Determine a non-default user name and password for accessing the bridges.
You should change the default user name and password.

c. Obtain an IP address, subnet mask, and gateway information for the Ethernet management 1
port on each bridge.

d. Disable VPN clients on the computer you are using for setup.
Active VPN clients cause the QuickNAV scan for bridges to fail.

Installing the FC-to-SAS bridge and SAS shelves

After ensuring that the system meets all the requirements in the Preparing for the installation section,
you can install your new system.

About this task

You should use an equal number of disk shelves at each site.

The system connectivity requirements for maximum distances for disk shelves, FC switches, and
backup tape devices using 50-micron, multimode fiber-optic cables, also apply to FibreBridges.
The Site Requirements Guide has detailed information about system connectivity requirements.

Note: SAS shelves in MetroCluster configurations do not require ACP cabling.

46 | MetroCluster Installation and Configuration Guide

Steps

1. Connect the Ethernet management 1 port on each bridge to your network using an Ethernet cable.
Note: The Ethernet management 1 port enables you to quickly download the bridge firmware
(using ATTO ExpressNAV or FTP management interfaces) and to retrieve core files and extract
logs.

2. Configure the Ethernet management 1 port for each bridge by following the procedure in the
ATTO FibreBridge 6500N Installation and Operation Manual, section 2.0.
Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet
management port that is connected by the Ethernet cable is configured. For example, if you
also wanted to configure the Ethernet management 2 port, you would need to connect the
Ethernet cable to port 2 and run QuickNAV.

3. Configure the bridges.

Be sure to make note of the user name and password that you designate.

a. Configure the bridge name.

The bridges should each have a unique name within the MetroCluster configuration.

b. Configure the data rate/speed of the bridge FC ports.

The most current information on supported distance can be found in the NetApp
Interoperability Matrix Tool.
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without
FlexArray). You use the Component Explorer to select the components and Data ONTAP
version to refine your search. You can click Show Results to display the list of supported
configurations that match the criteria.

c. Configure the connection mode that the bridges use to communicate across the FC network.
You must set the bridge connection mode to ptp (point-to-point).
For example, if you were to use the command line interface (CLI) to set the bridge FC 1 port's
basic required configuration, you would enter the following commands; the last command saves
the configuration changes:
set ipaddress
set subnet
set ipgateway
set FCDataRate 1 8Gb
set FCConnMode 1 ptp
set SNMP enabled
set bridgename
SaveConfiguration

Note: To set the IP address without the Quicknav utility, you need to have a serial connection
to the FibreBridge.

Result: You are prompted to restart the bridge.

The ATTO FibreBridge 6500N Installation and Operation Manual has the most current
information on available commands and how to use them.

4. Update the firmware on each bridge to the latest version by following the instructionsstarting
with Step 4on the FibreBridge 6500N Download page.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 47

5. Cable the disk shelves to the bridges by completing the following substeps:
Note: Do not force a connector into a port. The SAS cable QSFP connectors are keyed; when
oriented correctly into a SAS port, the QSFP connector clicks into place and the disk shelf SAS
port LNK LED illuminates green. For disk shelves, you insert a SAS cable connector with the
pull tab oriented down (on the underside of the connector).

a. Daisy-chain the disk shelves in each stack.

For information about daisy-chaining disk shelves, see the Installation and Service Guide for
your disk shelf model.
b. For each stack of disk shelves, cable IOM A square port of the first shelf to SAS port A on
FibreBridge A.

c. For each stack of disk shelves, cable IOM B circle port of the last shelf to SAS port A on
FibreBridge B.

Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack
through the first shelf, and bridge B connects to the B-side of the stack through the last shelf.
Note: The bridge SAS port B is disabled.

The following illustration shows a set of bridges cabled to a stack of three disk shelves:

FC1 FC2 FC1 FC2

FC_bridge_x_1_06 FC_bridge_x_2_o6

M1 SAS A M1 SAS A

Stack of SAS shelves

IOM A IOM B
First
shelf

Last
shelf

6. Verify that each bridge can detect all disk drives and disk shelves it is connected to.
48 | MetroCluster Installation and Configuration Guide

If you are using the... Then...

ATTO ExpressNAV GUI
a. In a supported web browser, enter the IP address of a bridge in the
browser box.
You are brought to the ATTO FibreBridge 6500N home page, which
has a link.

b. Click the link and enter your user name and the password that you
designated when you configured the bridge.
The ATTO FibreBridge 6500N status page appears with a menu to
the left.

c. Click Advanced in the menu.

d. Enter the following command, and then click Submit:

sastargets

Serial port connection Enter the following command:

sastargets

Example
The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines
are sequentially numbered so you can quickly count the devices. For example, the following
output shows that 10 disks are connected.

Tgt VendorID ProductID Type SerialNumber

0 NETAPP X410_S15K6288A15 DISK 3QP1CLE300009940UHJV
1 NETAPP X410_S15K6288A15 DISK 3QP1ELF600009940V1BV
2 NETAPP X410_S15K6288A15 DISK 3QP1G3EW00009940U2M0
3 NETAPP X410_S15K6288A15 DISK 3QP1EWMP00009940U1X5
4 NETAPP X410_S15K6288A15 DISK 3QP1FZLE00009940G8YU
5 NETAPP X410_S15K6288A15 DISK 3QP1FZLF00009940TZKZ
6 NETAPP X410_S15K6288A15 DISK 3QP1CEB400009939MGXL
7 NETAPP X410_S15K6288A15 DISK 3QP1G7A900009939FNTT
8 NETAPP X410_S15K6288A15 DISK 3QP1FY0T00009940G8PA
9 NETAPP X410_S15K6288A15 DISK 3QP1FXW600009940VERQ

Note: If the text response truncated appears at the beginning of the output, you can use
Telnet to connect to the bridge and enter the same command to see all the output.

7. Verify the command output shows the bridge is connected to all disks and disk shelves in the
stack that it is supposed to be connected to.

If the output is... Then...

Correct Repeat step 6 on page 47 for each remaining bridge.
Not correct
a. Check for loose SAS cables or correct the SAS cabling by repeating
step 5 on page 47.

b. Repeat step 6 on page 47.

8. Cable each bridge in the first switch fabric to the FC switches:

Note: The Brocade and Cisco switches use different port numbering:

On Brocade switches, the first port is numbered 0

On Cisco switches, the first port is numbered 1

This is reflected in the following table:

 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 49

Cable this bridge... To this port on FC_switch_x_1...

Brocade 6505 Brocade 6510 Cisco
bridge_A_1_port number 6 6 7
bridge_A_1_port number 7 7 8
bridge_A_1_port number - 8 9
bridge_A_1_port number - 9 10

9. Repeat the previous step on the bridges at the partner site.

Configuring the FC switches

For fabric-attached MetroCluster systems that were not pre-configured in the factory, you must
configure each FC switch in the DR group. This is done manually, or, depending on the switch, can
optionally be done with a configuration file.

About this task

For new systems, the FC switch fabrics are typically configured for two ISLs and do not require
additional configuration unless you want to change the pre-configured IP addresses.

Choices
Configuring the FC switches by running a configuration file on page 49
Configuring the Cisco or Brocade FC switches manually on page 50

Configuring the FC switches by running a configuration file

If you want to simplify the process of configuring switches, you can download and execute switch
configuration files that provide the complete switch settings for certain configurations.

About this task

You use the same configuration files for four-node and two-node configurations.

Choices
Configuring Brocade FC switches with configuration files on page 49
Configuring the Cisco FC switches with configuration files on page 50

Configuring Brocade FC switches with configuration files

When you configure a Brocade FC switch, you can download and execute switch configuration files
that provide the complete switch settings for certain configurations.

Before you begin

You must have access to an FTP server. The switches must have connectivity with the FTP server.

About this task

Each configuration file is different and must be used with the correct switch. Only one of the
configuration files for each switch fabric contains zoning commands.
50 | MetroCluster Installation and Configuration Guide

Steps

1. Go to the software download page.

NetApp Downloads: Software
2. In the list of products, find the row for Fibre Channel Switch, and in the drop-down list select
Brocade.

3. On the Fibre Channel Switch for Brocade page, click View & Download.

4. On the Fibre Channel Switch - Brocade page, click the MetroCluster link.

5. Follow the directions on the MetroCluster Configuration Files for Brocade Switches
description page to download and run the files.

Configuring the Cisco FC switches with configuration files

To configure a Cisco FC switch, you can download and execute switch configuration files that
provide the complete switch settings for certain configurations.

Steps

1. Go to the software download page.

NetApp Downloads: Software
2. In the list of products, find the row for Fibre Channel Switch, and then select Cisco from the
drop-down list.

3. On the Fibre Channel Switch for Cisco page, click the View & Download button.

4. On the Fibre Channel Switch - Cisco page, click the MetroCluster link.

5. Follow the directions on the MetroCluster Configuration Files for Cisco Switches Description
page to download and run the files.

Configuring the Cisco or Brocade FC switches manually

Switch configuration procedures and commands are different, depending on the switch vendor.

About this task

There are minor differences in switch configuration depending on whether you have a four-node or
two-node configuration. In two-node configurations, there is only one controller at each site. You can
ignore any steps that apply to ports for the second controller.

Choices
Configuring the Brocade FC switches on page 50
Configuring the Cisco FC switches on page 68

Configuring the Brocade FC switches

You must configure each of the Brocade switch fabrics in the MetroCluster configuration.

Before you begin

You must have a PC or UNIX workstation with Telnet or SSH access to the FC switches.

You must be using four supported Brocade switches of the same model with the same Brocade
Fabric Operating System (FOS) version and licensing.
NetApp Interoperability Matrix Tool
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 51

After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

You must have four switches; the MetroCluster configuration requires four switches.
The four switches must be connected to two fabrics of two switches each, with each fabric
spanning both sites.

Two initiator ports must be connected from each storage controller to each fabric.
Each storage controller must have four initiator ports available to connect to the switch fabrics.

About this task

ISL trunking is supported.

ISLs must have the same length and same speed ISLs in one fabric.
Different lengths can be used in the different fabrics. The same speed must be used in all fabrics.

Metro-E and TDM (SONET/SDH) are not supported; any non-FC native framing or signaling is
not supported.
Metro-E means Ethernet framing/signaling occurs either natively over a Metro distance or
through some TDM, MPLS, or WDM.

TDMs, FCR (native FC Routing) or FCIP extensions are not supported for the MetroCluster FC
switch fabric.

Third-party encryption devices are not supported on any link in the MetroCluster FC switch
fabric, including the ISL links across the WAN.

Certain switches in the MetroCluster FC switch fabric support encryption or compression, and
sometimes support both.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

The Brocade Virtual Fabric (VF) feature is not supported.

Steps
1. Reviewing Brocade license requirements on page 52
2. Setting the Brocade FC switch values to factory defaults on page 52
3. Configuring the basic switch settings on page 55
4. Configuring the E-ports on a Brocade FC switch on page 57
5. Configuring the non-E-ports on the Brocade switch on page 61
6. Configuring zoning on Brocade FC switches on page 62
7. Setting ISL encryption on Brocade 6510 switches on page 65

Related information
NetApp Interoperability Matrix Tool
52 | MetroCluster Installation and Configuration Guide

Reviewing Brocade license requirements

You need certain licenses for the switches in a MetroCluster configuration. You must install these
licenses on all four switches.
The MetroCluster configuration has the following Brocade license requirements:

Trunking license for systems using more than one ISL, as recommended.

Extended Fabric license (for ISL distances over 6 km)

Enterprise license for sites with more than one ISL and an ISL distance greater than 6 km
The Enterprise license includes Brocade Network Advisor and all licenses except for additional
port licenses.

You can verify that the licenses are installed by using the licenseshow command. If you do not
have these licenses, contact your sales representative before proceeding.

Setting the Brocade FC switch values to factory defaults

You must set the switch to its factory defaults to ensure a successful configuration. You must also
assign each switch a unique name.

About this task

In the examples in this procedure, the fabric consists of BrocadeSwitchA and BrocadeSwitchB.

Steps

1. Make a console connection and log in to both switches in one fabric.

2. Disable the switch persistently:

switchcfgpersistentdisable

This ensures the switch will remain disabled after a reboot or fastboot. If this command is not
available, use the switchdisable command.

Example
The following example shows the command on BrocadeSwitchA:

BrocadeSwitchA:admin> switchcfgpersistentdisable

The following example shows the command on BrocadeSwitchB:

BrocadeSwitchA:admin> switchcfgpersistentdisable

3. Enter switchname switch_name to set the switch name.

The switches should each have a unique name. After setting the name, the prompt changes
accordingly.

Example
The following example shows the command on BrocadeSwitchA:

BrocadeSwitchA:admin> switchname "FC_switch_A_1"

FC_switch_A_1:admin>

The following example shows the command on BrocadeSwitchB:

 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 53

BrocadeSwitchB:admin> switchname "FC_Switch_B_1"

FC_switch_B_1:admin>

4. Set all ports to their default values by issuing the following command for each port:
portcfgdefault

This must be done for all ports on the switch.

Example
The following example shows the commands on FC_switch_A_1:

FC_switch_A_1:admin> portcfgdefault 0
FC_switch_A_1:admin> portcfgdefault 1
...
FC_switch_A_1:admin> portcfgdefault 39

The following example shows the commands on FC_switch_B_1:

FC_switch_B_1:admin> portcfgdefault 0
FC_switch_B_1:admin> portcfgdefault 1
...
FC_switch_B_1:admin> portcfgdefault 39

5. Clear the zoning information by issuing the following commands:

cfgdisable
cfgclear
cfgsave

Example
The following example shows the commands on FC_switch_A_1:

FC_switch_A_1:admin> cfgdisable
FC_switch_A_1:admin> cfgclear
FC_switch_A_1:admin> cfgsave

The following example shows the commands on FC_switch_B_1:

FC_switch_B_1:admin> cfgdisable
FC_switch_B_1:admin> cfgclear
FC_switch_B_1:admin> cfgsave

6. Set the general switch settings to default:

configdefault

Example
The following example shows the command on FC_switch_A_1:

FC_switch_A_1:admin> configdefault

The following example shows the command on FC_switch_B_1:

FC_switch_B_1:admin> configdefault
54 | MetroCluster Installation and Configuration Guide

7. Set all ports to non-trunking mode:

switchcfgtrunk 0

Example
The following example shows the command on FC_switch_A_1:

FC_switch_A_1:admin> switchcfgtrunk 0

The following example shows the command on FC_switch_B_1:

FC_switch_B_1:admin> switchcfgtrunk 0

8. On Brocade 6510 switches, disable the Brocade Virtual Fabrics (VF) feature:
fosconfig options

Example
The following example shows the command on FC_switch_A_1:

FC_switch_A_1:admin> fosconfig --disable vf

The following example shows the command on FC_switch_B_1:

FC_switch_B_1:admin> fosconfig --disable vf

9. Clear the Administrative Domain (AD) configuration:

ad options

Example
The following example shows the commands on FC_switch_A_1:

FC_switch_A_1:admin> switch:admin> ad --select AD0

FC_switch_A_1:> defzone --noaccess
FC_switch_A_1:> cfgsave
FC_switch_A_1:> exit
FC_switch_A_1:admin> ad --clear -f
FC_switch_A_1:admin> ad --apply
FC_switch_A_1:admin> ad --save
FC_switch_A_1:admin> exit

The following example shows the commands on FC_switch_B_1:

FC_switch_B_1:admin> switch:admin> ad --select AD0

FC_switch_A_1:> defzone --noaccess
FC_switch_A_1:> cfgsave
FC_switch_A_1:> exit
FC_switch_B_1:admin> ad --clear -f
FC_switch_B_1:admin> ad --apply
FC_switch_B_1:admin> ad --save
FC_switch_B_1:admin> exit

10. Reboot the switch by issuing the following command:

reboot
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 55

Example
The following example shows the command on FC_switch_A_1:

FC_switch_A_1:admin> reboot

The following example shows the command on FC_switch_B_1:

FC_switch_B_1:admin> reboot

Configuring the basic switch settings

You must configure basic global settings, including the domain ID, for the Brocade switches.

About this task

This task contains steps that must be performed on each switch at both MetroCluster sites.
In this procedure, you set the unique domain ID for each switch as shown in the following examples:

FC_switch_A_1 is assigned to domain ID 5

FC_switch_A_2 is assigned to domain ID 6

FC_switch_B_1 is assigned to domain ID 7

FC_switch_B_2 is assigned to domain ID 8

Using that example, domain IDs 5 and 7 form fabric_1 and domain IDs 6 and 8 form fabric_2.

Steps

1. Enter configuration mode:

configure

2. Proceed through the prompts:

a. Set the domain ID for the switch.

b. Press Enter in response to the prompts until you get to RSCN Transmission Mode, and then
set that value to y.

c. Press Enter until you return to the switch prompt.

Example

FC_switch_A_1:admin> configure
Fabric parameters = y
Domain_id = 5
.
.
RSCN Transmission Mode (yes, y, no, n): [no] y

3. If you are using two or more ISLs per fabric, configure in-order-delivery of frames:
These steps must be performed on each switch fabric.

a. Enable in-order-delivery:
iodset

b. Set the Advanced Performance Tuning (APT) policy to 1:

aptpolicy 1
56 | MetroCluster Installation and Configuration Guide

c. Disable Dynamic Load Sharing (DLS):

dlsreset

d. Verify the IOD settings by using the iodshow, aptpolicy and dlsshow commands.

Example
For example, issue the following commands on FC_switch_A_1:

FC_switch_A_1:admin> iodshow
IOD is set

FC_switch_A_1:admin> aptpolicy
Current Policy: 1 0(ap)

3 0(ap) : Default Policy

1: Port Based Routing Policy
3: Exchange Based Routing Policy
0: AP Shared Link Policy
1: AP Dedicated Link Policy
command aptpolicy completed

FC_switch_A_1:admin> dlsshow
DLS is not set

e. Repeat these steps on the second switch fabric.

4. Enable the trap for T11-FC-ZONE-SERVER-MIB to provide successful health monitoring of the
switches in Data ONTAP:

a. Enable the T11-FC-ZONE-SERVER-MIB trap:

snmpconfig --set mibCapability -mib_name T11-FC-ZONE-SERVER-MIB

b. Repeat the previous step on the second switch fabric.

5. Reboot the switch:

reboot

Example
On FC_switch_A_1:

FC_switch_A_1:admin> reboot

On FC_switch_B_1:

FC_switch_B_1:admin> reboot

6. Persistently enable the switch:

switchcfgpersistentenable

Example
On FC_switch_A_1:

FC_switch_A_1:admin> switchcfgpersistentenable

On FC_switch_B_1:
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 57

FC_switch_B_1:admin> switchcfgpersistentenable

Configuring the E-ports on a Brocade FC switch

On each switch fabric, you must configure the switch ports that connect the Inter-Switch Link (ISL).
These ISL ports are also known as the E-ports.

Before you begin

All ISLs in an FC switch fabric must be configured with the same speed and distance.
The combination of the switch port and SFP must support the speed.
The ISL must be using one of the supported speeds: 4 Gbps, 8 Gbps, or 16 Gbps.

The distance can be as far as 200 km but must be supported by the FC switch model.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

The ISL link must have a dedicated lambda, and the link must be supported by Brocade for the
distance, switch type, and FOS.

About this task

You must not use the L0 setting when issuing the portCfgLongDistance command. Instead, you
should use the LE or LS setting to configure the distance on the Brocade switches with a minimum of
LE.
You must not use the LD setting when issuing the portCfgLongDistance command when working
with xWDM/TDM equipment. Instead, you should use the LE or LS setting to configure the distance
on the Brocade switches.
You must perform this task for each FC switch fabric.
The following tables show the ISL ports for the different switches and different numbers of ISLs:

Ports for dual ISL configurations (all switches)

FC_switch_x_1 ISL ports FC_switch_B_1 ports
10 10
11 11

Ports for three or four ISL configurations (Brocade 6505)

FC_switch_A_1 ISL ports FC_switch_B_1 ports
8 8
9 9
10 10
11 11
58 | MetroCluster Installation and Configuration Guide

Ports for three or four ISL configurations (Brocade 6510)

FC_switch_A_1 ISL ports FC_switch_B_1 ports
20 20
21 21
22 22
23 23

Steps

1. Configure the port speed:

portcfgspeed port-number speed

You must use the highest common speed supported by the components in the path.

Example
In the following example, there is one ISL for each fabric:

FC_switch_A_1:admin> portcfgspeed 10 16

FC_switch_B_1:admin> portcfgspeed 10 16

In the following example, there are two ISLs for each fabric:

FC_switch_A_1:admin> portcfgspeed 10 16
FC_switch_A_1:admin> portcfgspeed 11 16

FC_switch_B_1:admin> portcfgspeed 10 16
FC_switch_B_1:admin> portcfgspeed 11 16

2. If more than one ISL for each fabric is used, enable trunking for each ISL port:
portcfgtrunkport port-number 1

Example

FC_switch_A_1:admin> portcfgtrunkport 10 1
FC_switch_A_1:admin> portcfgtrunkport 11 1

FC_switch_B_1:admin> portcfgtrunkport 10 1
FC_switch_B_1:admin> portcfgtrunkport 11 1

3. Enable QoS traffic for each of the ISL ports:

portcfgqos --enable port-number

Example
In the following example, there is one ISL per switch fabric:

FC_switch_A_1:admin> portcfgqos --enable 10

FC_switch_B_1:admin> portcfgqos --enable 10

Example
In the following example, there are two ISLs per switch fabric:
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 59

FC_switch_A_1:admin> portcfgqos --enable 10

FC_switch_A_1:admin> portcfgqos --enable 11

FC_switch_B_1:admin> portcfgqos --enable 10

FC_switch_B_1:admin> portcfgqos --enable 11

4. Verify the settings by using the portCfgShow command.

Example
The following example shows the output for a configuration that uses two ISLs cabled to port 10
and port 11:

Ports of Slot 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
----------------+---+---+---+---+-----+---+---+---+----+---+---+---+-----+---+---+---
Speed AN AN AN AN AN AN 8G AN AN AN 16G 16G AN AN AN AN
Fill Word 0 0 0 0 0 0 3 0 0 0 3 3 3 0 0 0
AL_PA Offset 13 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Trunk Port .. .. .. .. .. .. .. .. .. .. ON ON .. .. .. ..
Long Distance .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
VC Link Init .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked L_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked G_Port .. .. .. .. .. .. ON .. .. .. .. .. .. .. .. ..
Disabled E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
ISL R_RDY Mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
RSCN Suppressed .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Persistent Disable.. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
LOS TOV enable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
NPIV capability ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
NPIV PP Limit 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126
QOS E_Port AE AE AE AE AE AE AE AE AE AE AE AE AE AE AE AE
Mirror Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Rate Limit .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Credit Recovery ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
Fport Buffers .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Port Auto Disable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
CSCTL mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..

Fault Delay 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

5. Calculate the ISL distance.

Due to the behavior of FC-VI, the distance must be set to 1.5 times the real distance with a
minimum of 10 (using the LE distance level).
The distance for the ISL is calculated as follows, rounded up to the next full kilometer:
1.5 x real_distance = distance

Example
The distance is 3 km, then 1.5 x 3km = 4.5. This is lower than 10, so the ISL must be set to the
LE distance level.

Example
The distance is 20 km, then 1.5 x 20km = 30. The ISL must be set to 30. and use the LS distance
level.

6. Set the distance on each ISL port:

portcfglongdistance
port distance-level vc_link_init distance

A vc_link_init value of 1 uses the ARB fill word (default). A value of 0 uses IDLE. The
required value might depend on the link being used. The commands must be repeated for each
ISL port.
60 | MetroCluster Installation and Configuration Guide

Example
For an ISL distance of 3 km, as given in the example in the previous step, the setting is 4.5 with
the default vc_link_init value of 1:

FC_switch_A_1:admin> portcfglongdistance 10 LE 1 4.5

FC_switch_B_1:admin> portcfglongdistance 10 LE 1 4.5

Example
For an ISL distance of 20 km, as given in the example in the previous step, the setting is 30 with
the default vc_link_init value of 1:

FC_switch_A_1:admin> portcfglongdistance 10 LS 1 30

FC_switch_B_1:admin> portcfglongdistance 10 LS 1 30

7. Verify the distance setting:

portbuffershow

A distance level of LE appears as 10km.

Example
The following example shows output is a configuration that uses ISLs on port 10 and port 11:

FC_switch_A_1:admin> portbuffershow

User Port Lx Max/Resv Buffer Needed Link Remaining

Port Type Mode Buffers Usage Buffers Distance Buffers
---- ---- ---- ------- ------ ------- --------- ---------
...
10 E - 8 67 67 30km
11 E - 8 67 67 30km
...
23 - 8 0 - - 466

8. Verify that both switches form one fabric:

switchshow

Example
The following example shows the output for a configuration that uses ISLs on port 10 and port
11:

FC_switch_A_1:admin> switchshow
switchName: FC_switch_A_1
switchType: 71.2
switchState:Online
switchMode: Native
switchRole: Subordinate
switchDomain: 5
switchId: fffc01
switchWwn: 10:00:00:05:33:86:89:cb
zoning: OFF
switchBeacon: OFF

Index Port Address Media Speed State Proto

===========================================
...
10 10 010C00 id 16G Online FC E-Port 10:00:00:05:33:8c:2e:9a "FC_switch_B_1"
11 11 010B00 id 16G Online FC E-Port 10:00:00:05:33:8c:2e:9a "FC_switch_B_1"
(upstream)
...

FC_switch_B_1:admin> switchshow
switchName: FC_switch_B_1
switchType: 71.2
switchState:Online
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 61

switchMode: Native
switchRole: Principal
switchDomain: 7
switchId: fffc03
switchWwn: 10:00:00:05:33:8c:2e:9a
zoning: OFF
switchBeacon: OFF

Index Port Address Media Speed State Proto

==============================================
...
10 10 030A00 id 16G Online FC E-Port 10:00:00:05:33:86:89:cb "FC_switch_A_1"
11 11 030B00 id 16G Online FC E-Port 10:00:00:05:33:86:89:cb "FC_switch_A_1"
(downstream)
...

9. Confirm the configuration of the fabrics:

fabricshow

Example

FC_switch_A_1:admin> fabricshow
Switch ID Worldwide Name Enet IP Addr FC IP Addr Name
-----------------------------------------------------------------
1: fffc01 10:00:00:05:33:86:89:cb 10.10.10.55 0.0.0.0 "FC_switch_A_1"
3: fffc03 10:00:00:05:33:8c:2e:9a 10.10.10.65 0.0.0.0 >"FC_switch_B_1"

FC_switch_B_1:admin> fabricshow
Switch ID Worldwide Name Enet IP Addr FC IP Addr Name
----------------------------------------------------------------
1: fffc01 10:00:00:05:33:86:89:cb 10.10.10.55 0.0.0.0 "FC_switch_A_1"

3: fffc03 10:00:00:05:33:8c:2e:9a 10.10.10.65 0.0.0.0 >"FC_switch_B_1

10. Repeat the previous steps for the second FC switch fabric.

Related concepts
Port assignments for FC switches in a four-node configuration on page 38

Configuring the non-E-ports on the Brocade switch

You must configure the non-E-ports on the FC switch. In the MetroCluster configuration, these are
the ports that connect the switch to the HBA initiators, FC-VI interconnects, and FC-to-SAS bridges.
These steps must be done for each port.

About this task

In the following example, the ports connect an FC-to-SAS bridge:

Port 6 on FC_FC_switch_A_1 at Site_A

Port 6 on FC_FC_switch_B_1 at Site_B

Steps

1. Configure the port speed for each non-E-port:

portcfgspeed port speed

You should use the highest common speed, which is the highest speed supported by all
components in the data path: the SFP, the switch port that the SFP is installed on, and the
connected device (HBA, bridge, etc).
For example, the components might have the following supported speeds:

The SFP is capable of 1/2/4 GB.

The switch port is capable of 1/2/4/8 GB.

62 | MetroCluster Installation and Configuration Guide

The connected HBA maximum speed is 4 GB.

The highest common speed in this case is 4 GB, so the port should be configured for a speed of 4
GB.

Example

FC_switch_A_1:admin> portcfgspeed 6 4

FC_switch_B_1:admin> portcfgspeed 6 4

2. Verify the settings using the portcfgshow command:

Example

FC_switch_A_1:admin> portcfgshow

FC_switch_B_1:admin> portcfgshow

In the example output, port 6 has the following settings:

Speed is set to 4G

Ports of Slot 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
-----------------+---+---+---+---+-----+---+---+---+-----+---+---+---+-----+---+---+---
Speed AN AN AN AN AN AN 4G AN AN AN AN AN AN AN AN AN
Fill Word 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0
AL_PA Offset 13 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Trunk Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Long Distance .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
VC Link Init .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked L_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked G_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Disabled E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Locked E_Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
ISL R_RDY Mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
RSCN Suppressed .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Persistent Disable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
LOS TOV enable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
NPIV capability ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
NPIV PP Limit 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126
QOS E_Port AE AE AE AE AE AE AE AE AE AE AE AE AE AE AE AE
Mirror Port .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Rate Limit .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Credit Recovery ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON ON
Fport Buffers .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
Port Auto Disable .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..
CSCTL mode .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. ..

Fault Delay 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Configuring zoning on Brocade FC switches

You must assign the switch ports to separate zones to separate controller and storage traffic, one zone
for the FC-VI ports and one for the storage ports.

About this task

For the FC-VI ports, a Quality of Service (QoS) zone is required. A QoS zone has a special name to
differentiate it from a regular zone. The FC-VI zone must have the name start with the prefix
QOSH1_, followed by a user-defined portion of the name.

Zone type Contains Priority Example names

FC-VI Four ports, one for Always high priority QOSH1_FCVI_1
each FC-VI cable
from each controller
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 63

Zone type Contains Priority Example names

Storage Nine ports: Does not apply STOR_A_1_6
Storage ports use
Eight HBA
standard zoning.
initiator ports (two
HBA connections
for each
controller).

One port
connecting to an
FC-to-SAS bridge.

Note: Zoning for the fabric can be configured from one switch in the fabric. In this example, it is
configured on Switch_A_1.

The examples in the following steps use these ports and zones:

Zone Switch Domain Port Port usage

QOSH1_FCVI_1 5 0 FC-VI
3
7 0
3
STOR_A_1_6 5 1 HBA
2
4
5
6 ATTO
STOR_B_1_6 7 1 HBA
2
4
5
6 ATTO

Steps

1. Create the FC-VI QoS zone by using the zonecreate command:

zonecreate "QOSH1_FCVI_1", member;member ...

Example
In this example, ports 0 and 3 of domain 1 (Switch_A_1) and ports 0 and 4 of domain 3
(Switch_B_1) are members of the FC-VI zone.

Switch_A_1:admin> zonecreate "QOSH1_FCVI_1", "5,0; 5,3; 7,0; 7,3"

2. Create the storage zone for switch domains:

zonecreate name, member;member ...
64 | MetroCluster Installation and Configuration Guide

Example
In the following example, ports 1, 2, 4 and 5 of domain 5 (Switch_A_1) connect with the HBAs
on the storage controllers. Port 6 in domain 5 (Switch_A_1) connects to an FC-to-SAS bridge.
Note: You should give each zone a descriptive name. In this example, it is "STOR_A_1_6" that
identifies the zone as a storage zone for the target port 6 at Site_A.

Switch_A_1:admin> zonecreate "STOR_A_1_6", "5,1; 5,2; 5,4; 5,5; 7,1;

7,2; 7,4; 7,5; 7,6"
Switch_A_1:admin> zonecreate "STOR_B_1_6", "5,1; 5,2; 5,4; 5,5; 7,1;
7,2; 7,4; 7,5; 7,6"

3. Create the configuration: cfgcreate config_name , zone;zone...

Example

Switch_A_1:admin> cfgcreate "CFG_1", "QOSH1_FCVI_1; STOR_A_1_6;

STOR_B_1_6"

4. Enter the cfgadd config_name zone;zone... command if you want to add more zones to the
configuration.

5. Enable the configuration: cfgenable config_name

Example

Switch_A_1:admin> cfgenable "CFG_1"

6. Save the configuration: cfgsave

Example

Switch_A_1:admin> cfgsave

7. Verify the zoning configuration: zone --validate

Example

Switch_A_1:admin> zone --validate

Defined configuration:
cfg: CFG_1 QOSH1_FCVI_1; STOR_A_1_6; STOR_B_1_6
zone: QOSH1_FCVI_1
5,0; 7,0; 5,3; 7,3
zone: STOR_A_1_6
5,1; 5,2; 5,4; 5,5; 7,1; 7,2; 7,4; 7,5; 5,6
zone: storage_zone_5_7
5,1; 5,2; 5,4; 5,5; 7,1; 7,2; 7,4; 7,5; 7,6

Effective configuration:
cfg: CFG_1
zone: QOSH1_FCVI_1
5,0
7,0
5,3
7,3
zone: STOR_A_1_6
5,1
5,2
5,4
5,5
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 65

7,1
7,2
7,4
7,5
5,6
zone: STOR_B_1_6
5,1
5,2
5,4
5,5
7,1
7,2
7,4
7,5
7,6
------------------------------------
~ - Invalid configuration
* - Member does not exist
# - Invalid usage of broadcast zone

After you finish

You must configure ISL encryption, or, if you are not using ISL encryption, repeat the configuration
tasks on the second FC switch fabric.

Setting ISL encryption on Brocade 6510 switches

On Brocade 6510 switches, you can optionally use the Brocade encryption feature on the ISL
connections. If you want to use the encryption feature, you must perform additional configuration
steps on each switch in the MetroCluster configuration.

Before you begin

You must have Brocade 6510 switches.

You must have selected two switches from the same fabric.

About this task

The steps must be performed on both the switches in the same fabric.

Disabling virtual fabric

In order to set the ISL encryption, you must disable the virtual fabric on all the four switches being
used in a MetroCluster configuration.

Step

1. Disable the virtual fabric by entering the following command at the switch console:
fosconfig --disable vf

After you finish

Reboot the switch.

Setting the payload

After disabling the virtual fabric, you must set the payload or the data field size on both switches in
the fabric.

About this task

The data field size must not exceed 2048.
66 | MetroCluster Installation and Configuration Guide

Steps

1. Disable the switch:

switchdisable

2. Configure and set the payload:

configure

3. Set the following switch parameters:

a. Set the Fabric parameter as follows:

y

b. Set the other parameters such as Domain, WWN Based persistent PID and so on.

c. Set the data field size as follows:

2048

4. Enable the switch:

switchenable

Setting the authentication policy

You must set the authentication policy, type, group and secret.

About this task

The commands must be executed at the switch console.

Steps

1. Disable the switch:

switchdisable

2. Set the authentication policy on the switch to on by entering the following command:
authUtil --policy -sw on

The system displays the following output:

Warning: Activating the authentication policy requires either DH-CHAP

secrets or PKI certificates depending on the protocol selected.
Otherwise, ISLs will be segmented during next E-port bring-up.
ARE YOU SURE (yes, y, no, n): [no] yes
Auth Policy is set to ON

3. Set the authentication type to dhchap by entering the following command:

authUtil --set -a dhchap

The system displays the following output:

Authentication is set to dhchap.

4. Set the authentication group to 4 by entering the following command:

authUtil --set -g 4

5. Set the authentication secret by performing the following steps:

a. Provide the worldwide name (WWN) of the other switch in the fabric for the parameter
Enter peer WWN, Domain, or switch name.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 67

b. Provide the peer secret for the parameter Enter peer secret.

c. Provide the local secret for the parameter Enter local secret.

d. Enter the following value for the parameter Are you done.
Y

Example
The following is an example of setting authentication secret:

brcd> secAuthSecret --set

This command is used to set up secret keys for the DH-CHAP

authentication.
The minimum length of a secret key is 8 characters and maximum 40
characters. Setting up secret keys does not initiate DH-CHAP
authentication. If switch is configured to do DH-CHAP, it is performed
whenever a port or a switch is enabled.

Warning: Please use a secure channel for setting secrets. Using

an insecure channel is not safe and may compromise secrets.

Following inputs should be specified for each entry.

1. WWN for which secret is being set up.

2. Peer secret: The secret of the peer that authenticates to peer.
3. Local secret: The local secret that authenticates peer.

Press enter to start setting up secrets > <cr>

Enter peer WWN, Domain, or switch name (Leave blank when done):
10:00:00:05:33:76:2e:99
Enter peer secret: <hidden>
Re-enter peer secret: <hidden>
Enter local secret: <hidden>
Re-enter local secret: <hidden>

Enter peer WWN, Domain, or switch name (Leave blank when done):
Are you done? (yes, y, no, n): [no] yes
Saving data to key store... Done.

6. Enable the switch:

switchenable

Enabling ISL encryption in a Brocade switch

After setting the authentication policy and the authentication secret, you must enable ISL encryption
on the Brocade 6510 switches.

About this task

These steps should be performed on one switch fabric at a time.

The commands must be run at the switch console.

Steps

1. Disable the switch:

switchdisable

2. Enable encryption on all the ISL ports:

portCfgEncrypt --enable port_number
68 | MetroCluster Installation and Configuration Guide

Example
In the following example, the encryption is enabled on ports 8 and 12:
portCfgEncrypt --enable 8
portCfgEncrypt --enable 12

3. Enable the switch:

switchenable

4. Verify that the ISL is up and working:

islshow

5. Verify that the encryption is enabled:

portenccompshow

Example
The following example shows the encryption is enabled on ports 8 and 12:

User Encryption
Port configured Active
---- ---------- ------
8 yes yes
9 No No
10 No No
11 No No
12 yes yes

After you finish

Perform all the steps on the switches in the other fabric in a MetroCluster configuration.

Configuring the Cisco FC switches

You must ensure that the storage and ISL connection requirements are met and configure each Cisco
switch in the MetroCluster configuration.

About this task

The following requirements apply to the Cisco FC switches:

You must be using four supported Cisco switches of the same model with the same NX-OS
version and licensing.

The MetroCluster configuration requires four switches.

The four switches must be connected into two fabrics of two switches each, with each fabric
spanning both sites.

Not all switches are supported for connectivity to the ATTO FiberBridge model.

Encryption and compression in the Cisco FC storage fabric is not supported in the MetroCluster
configuration.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray). You
use the Component Explorer to select the components and Data ONTAP version to refine your
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 69

search. You can click Show Results to display the list of supported configurations that match the
criteria.
The following requirements apply to the Inter-Switch Link (ISL) connections:

Fibre Channel over IP (FCIP) is not supported for ISL connections in a MetroCluster
environment.

ISLs of different speeds and lengths are supported between switches in the same fabric.

The following requirement applies to the storage connections:

Two initiator ports must be connected from each storage controller to each fabric.
Each storage controller must have four initiator ports available to connect to the switch fabrics.

Steps
1. Reviewing Cisco license requirements on page 69
2. Setting the Cisco FC switch to factory defaults on page 70
3. Configure the Cisco FC switch basic settings and community string on page 70
4. Acquiring licenses for ports on page 71
5. Enabling ports in a Cisco MDS 9148 or 9148S switch on page 72
6. Configuring the F-ports on a Cisco FC switch on page 73
7. Assigning buffer-to-buffer credits to F-Ports in the same port group as the ISL on page 74
8. Creating and configuring the VSANs on Cisco FC switches on page 76
9. Configuring the E-ports on the Cisco FC switch on page 80
10. Configuring zoning on a Cisco FC switch on page 83
11. Ensuring the FC switch configuration is saved on page 86

Related information
NetApp Interoperability Matrix Tool

Reviewing Cisco license requirements

Certain feature-based licenses might be required for the switches in a MetroCluster configuration.
These licenses enable you to use features such as QoS or long-distance mode credits on the switches.
You must install these licenses on all four switches in a MetroCluster configuration.
The following are the feature-based licenses that might be required in a MetroCluster configuration:

ENTERPRISE_PKG
This enables you to use the QoS feature in Cisco switches.

PORT_ACTIVATION_PKG
You can use this license for Cisco 9148 switches. This license enables you to activate or
deactivate ports on the switches as long as only 16 ports are active at any given time. By default,
16 ports are enabled in Cisco MDS 9148 switches.

FM_SERVER_PKG
This enables you to manage fabrics simultaneously and to manage switches through a web
browser.
The FM_SERVER_PKG license also enables performance management features such as
performance thresholds, threshold monitoring, and so on. For more information about this license,
see the Cisco Fabric Manager Server Package.

You can verify that the licenses are installed by using the show license usage command. If you
do not have these licenses, contact your sales representative before proceeding.
70 | MetroCluster Installation and Configuration Guide

Setting the Cisco FC switch to factory defaults

To ensure a successful configuration, you must set the switch to its factory defaults. This ensures that
the switch is starting from a clean configuration.

About this task

This task must be performed on all switches in the MetroCluster configuration.

Steps

1. Make a console connection and log in to both switches in the same fabric.

2. Issue the following command to set the switch back to its default settings:
write erase

You can respond y when prompted to confirm the command. This erases all licenses and
configuration information on the switch.

3. Issue the following command to reboot the switch:

reload

You can respond y when prompted to confirm the command.

4. Repeat the write erase and reload commands on the other switch.
After issuing the reload command, the switch reboots and then prompts with setup questions. At
that point, proceed to the next section.

The following example shows the process on a fabric consisting of FC_switch_A_1 and
FC_switch_B_1.

FC_Switch_A_1# write erase

Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n) [n] y
FC_Switch_A_1# reload
This command will reboot the system. (y/n)? [n] y

FC_Switch_B_1# write erase

Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n) [n] y
FC_Switch_B_1# reload
This command will reboot the system. (y/n)? [n] y

Configure the Cisco FC switch basic settings and community string

You must specify the basic settings with the setup command or after issuing the reload command.

Steps

1. If the switch does not display the setup questions, issue the following command to configure the
basic switch settings:
setup

2. Accept the default responses to the setup questions until you are prompted for the SNMP
community string.

3. Set the community string to public (all lowercase) to allow access from the Data ONTAP Health
Monitors.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 71

Example
The following example shows the commands on FC_switch_A_1:

FC_switch_A_1# setup
Configure read-only SNMP community string (yes/no) [n]: y
SNMP community string : public
Note: Please set the SNMP community string to "Public" or
another value of your choosing.
Configure default switchport interface state (shut/noshut)
[shut]: noshut
Configure default switchport port mode F (yes/no) [n]: n
Configure default zone policy (permit/deny) [deny]: deny
Enable full zoneset distribution? (yes/no) [n]: yes

The following example shows the commands on FC_switch_B_1:

FC_switch_B_1# setup
Configure read-only SNMP community string (yes/no) [n]: y
SNMP community string : public
Note: Please set the SNMP community string to "Public" or
another value of your choosing.
Configure default switchport interface state (shut/noshut)
[shut]: noshut
Configure default switchport port mode F (yes/no) [n]: n
Configure default zone policy (permit/deny) [deny]: deny
Enable full zoneset distribution? (yes/no) [n]: yes

Acquiring licenses for ports

You do not have to use Cisco switch licenses on a continuous range of ports; instead, you can acquire
licenses for specific ports that are used and remove licenses from unused ports. You should verify the
number of licensed ports in the switch configuration and, if necessary, move licenses from one port to
another as needed.

Steps

1. Issue the following command to show license usage for a switch fabric:
show port-resources module 1

Determine which ports require licenses. If some of those ports are unlicensed, determine if you
have extra licensed ports and consider removing the licenses from them.

2. Issue the following command to enter configuration mode:

config t

3. Remove the license from the selected port:

a. Issue the following command to select the port to be unlicensed:

interface interface-name

b. Remove the license from the port using the following command:
no port-license acquire

c. Exit the port configuration interface:

exit

4. Acquire the license for the selected port:

a. Issue the following command to select the port to be unlicensed:

interface interface-name
72 | MetroCluster Installation and Configuration Guide

b. Make the port eligible to acquire a license using the "port license" command:
port-license

c. Acquire the license on the port using the following command:

port-license acquire

d. Exit the port configuration interface:

exit

5. Repeat for any additional ports.

6. Issue the following command to exit configuration mode:

exit

Removing and acquiring a license on a port

This example shows a license being removed from port fc1/2, port fc1/1 being made eligible to
acquire a license, and the license being acquired on port fc1/1:

Switch_A_1# conf t
Switch_A_1(config)# interface fc1/2
Switch_A_1(config)# shut
Switch_A_1(config-if)# no port-license acquire
Switch_A_1(config-if)# exit
Switch_A_1(config)# interface fc1/1
Switch_A_1(config-if)# port-license
Switch_A_1(config-if)# port-license acquire
Switch_A_1(config-if)# no shut
Switch_A_1(config-if)# end
Switch_A_1# copy running-config startup-config

Switch_B_1# conf t
Switch_B_1(config)# interface fc1/2
Switch_B_1(config)# shut
Switch_B_1(config-if)# no port-license acquire
Switch_B_1(config-if)# exit
Switch_B_1(config)# interface fc1/1
Switch_B_1(config-if)# port-license
Switch_B_1(config-if)# port-license acquire
Switch_B_1(config-if)# no shut
Switch_B_1(config-if)# end
Switch_B_1# copy running-config startup-config

The following example shows port license usage being verified:

Switch_A_1# show port-resources module 1

Switch_B_1# show port-resources module 1

Enabling ports in a Cisco MDS 9148 or 9148S switch

In Cisco MDS 9148 or 9148S switches, you must manually enable the ports required in a
MetroCluster configuration.

About this task

You can manually enable 16 ports in a Cisco MDS 9148 or 9148S switch.

The Cisco switches enable you to apply the POD license on random ports, as opposed to applying
them in sequence.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 73

Cisco switches require that you use one port from each port group, unless you need more than 12
ports.

Steps

1. View the port groups available in a Cisco switch:

show port-resources module blade_number

2. License and acquire the required port in a port group by entering the following commands in
sequence:
config t
interface port_number
shut
port-license acquire
no shut

Example
For example, the following command licenses and acquires Port fc 1/45:

switch# config t
switch(config)#
switch(config)# interface fc 1/45
switch(config-if)#
switch(config-if)# shut
switch(config-if)# port-license acquire
switch(config-if)# no shut
switch(config-if)# end

3. Save the configuration:

copy running-config startup-config

Configuring the F-ports on a Cisco FC switch

You must configure the F-ports on the FC switch. In a MetroCluster configuration, the F-ports are the
ports that connect the switch to the HBA initiators, FC-VI interconnects and FC-to-SAS bridges.
Each port must be configured individually.

About this task

The following table lists the ports that must be configured as F-ports (switch-to-node) and shows
what each port connects to:

Configure this port to F-mode: Port connects to...

1 controller_x_1 FC-VI port 1
2 controller_x_1 HBA port 1
3 controller_x_1 HBA port 2
4 controller_x_2 FC-VI port 1
5 controller_x_2 HBA 1
6 controller_x_2 HBA 2
7 FC-to-SAS bridge

This task must be performed on each switch in the MetroCluster configuration.

74 | MetroCluster Installation and Configuration Guide

Steps

1. Issue the following command to enter configuration mode:

config t

2. Enter interface configuration mode for the port:

interface port-ID

3. Shut down the port:

shutdown

4. Set the ports to F mode by issuing the following command:

switchport mode F

5. Set the ports to fixed speed by issuing the following command:

switchport speed speed
speed is either 8000 or 16000

6. Set the rate mode of the switch port to dedicated by issuing the following command:
switchport rate-mode dedicated

7. Restart the port:

no shutdown

8. Issue the following command to exit configuration mode:

end

The following example shows the commands on the two switches:

Switch_A_1# config t
FC_switch_A_1(config)# interface fc 1/1
FC_switch_A_1(config-if)# shutdown
FC_switch_A_1(config-if)# switchport mode F
FC_switch_A_1(config-if)# switchport speed 8000
FC_switch_A_1(config-if)# switchport rate-mode dedicated
FC_switch_A_1(config-if)# no shutdown
FC_switch_A_1(config-if)# end
FC_switch_A_1# copy running-config startup-config

FC_switch_B_1# config t
FC_switch_B_1(config)# interface fc 1/1
FC_switch_B_1(config-if)# switchport mode F
FC_switch_B_1(config-if)# switchport speed 8000
FC_switch_B_1(config-if)# switchport rate-mode dedicated
FC_switch_B_1(config-if)# no shutdown
FC_switch_B_1(config-if)# end
FC_switch_B_1# copy running-config startup-config

Assigning buffer-to-buffer credits to F-Ports in the same port group as the ISL
You must assign the buffer-to-buffer credits to the F-ports if they are in the same port group as the
ISL. If the ports do not have the required buffer-to-buffer credits, the ISL could be inoperative. This
task is not required if the F-ports are not in the same port group as the ISL port.

About this task

If the F-Ports are in a port group that contains the ISL, this task must be performed on each FC
switch in the MetroCluster configuration.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 75

Steps

1. Issue the following command to enter configuration mode:

config t

2. Enter the following command to set the interface configuration mode for the port:
interface port-ID

3. Disable the port:

shut

4. If the port is not already in F mode, set the port to F mode by entering the following command:
switchport mode F

5. Set the buffer-to-buffer credit of the non-E ports to 1 by using the following command:
switchport fcrxbbcredit 1

6. Re-enable the port:

no shut

7. Exit configuration mode:

exit

8. Copy the updated configuration to the startup configuration:

copy running-config startup-config

9. Verify the buffer-to-buffer credit assigned to a port by entering the following commands:
show port-resources module 1

10. Issue the following command to exit configuration mode:

exit

11. Repeat these steps on the other switch in the fabric.

12. Verify the settings:

show port-resource module 1

In this example, port fc1/40 is the ISL. Ports fc1/37, fc1/38 and fc1/39 are in the same port
group and must be configured.
The following commands show the port range being configured for fc1/37 through fc1/39:

FC_switch_A_1# conf t
FC_switch_A_1(config)# interface fc1/37-39
FC_switch_A_1(config-if)# shut
FC_switch_A_1(config-if)# switchport mode F
FC_switch_A_1(config-if)# switchport fcrxbbcredit
1FC_switch_A_1(config-if)# no shut
FC_switch_A_1(config-if)# exit
FC_switch_A_1# copy running-config startup-config

FC_switch_B_1# conf t
FC_switch_B_1(config)# interface fc1/37-39
FC_switch_B_1(config-if)# shut
FC_switch_B_1(config-if)# switchport mode F
FC_switch_B_1(config-if)# switchport fcrxbbcredit 1
FC_switch_A_1(config-if)# no shut
FC_switch_A_1(config-if)# exit
FC_switch_B_1# copy running-config startup-config
76 | MetroCluster Installation and Configuration Guide

The following commands and system output show that the settings are properly applied:

FC_switch_A_1# show port-resource module 1

...
Port-Group 11
Available dedicated buffers are 93

--------------------------------------------------------------------
Interfaces in the Port-Group B2B Credit Bandwidth Rate
Mode
Buffers (Gbps)

--------------------------------------------------------------------
fc1/37 32 8.0 dedicated
fc1/38 1 8.0 dedicated
fc1/39 1 8.0 dedicated
...

FC_switch_B_1# port-resource module

...
Port-Group 11
Available dedicated buffers are 93

--------------------------------------------------------------------
Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode
Buffers (Gbps)

--------------------------------------------------------------------
fc1/37 32 8.0 dedicated
fc1/38 1 8.0 dedicated
fc1/39 1 8.0 dedicated
...

Creating and configuring the VSANs on Cisco FC switches

You must create a VSAN for the FC-VI ports and a VSAN for the storage ports on each FC switch in
the MetroCluster configuration. The VSANs should have a unique number and name. You must do
additional configuration if you are using two ISLs with in-order delivery of frames.

About this task

The examples here use the following naming conventions:

Switch fabric VSAN name ID number

1 FCVI_1_10 10
STOR_1_20 20
2 FCVI_2_30 30
STOR_2_20 40

This task must be performed on each FC switch fabric.

Steps

1. Configure the FC-VI VSAN:

a. Issue the following command to enter configuration mode if you have not done so already:
config t

b. Issue the following command to edit the VSAN database:

vsan database
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 77

c. Set the VSAN ID using the following command:

vsan vsan-ID

d. Set the VSAN name using the following command:

vsan vsan-ID name vsan_name

Example
The following example shows the commands on FC_switch__A_1:

FC_switch_A_1# conf t
FC_switch_A_1(config)# vsan database
FC_switch_A_1(config-vsan-db)# vsan 10
FC_switch_A_1(config-vsan-db)# vsan 10 name FCVI_1_10
FC_switch_A_1(config-vsan-db)# end
FC_switch_A_1# copy running-config startup-config

The following example shows the commands on FC_switch__B_1:

FC_switch_B_1# conf t
FC_switch_B_1(config)# vsan database
FC_switch_B_1(config-vsan-db)# vsan 10
FC_switch_B_1(config-vsan-db)# vsan 10 name FCVI_1_10
FC_switch_B_1(config-vsan-db)# end
FC_switch_B_1# copy running-config startup-config

2. Add ports to the FC-VI VSAN:

a. Issue the following command for each port in the VSAN:

vsan vsan-ID interface interface_name

For the FC-VI VSAN, the ports connecting the two local FC-VI ports will be added. In the
following example, the port s are fc1/1 and fc1/13:

Example

FC_switch_A_1# conf t
FC_switch_A_1(config)# vsan database
FC_switch_A_1(config)# vsan 10 interface fc1/1
FC_switch_A_1(config)# vsan 10 interface fc1/13
FC_switch_A_1(config)# end
FC_switch_A_1# copy running-config startup-config
FC_switch_B_1# conf t
FC_switch_B_1(config)# vsan database
FC_switch_B_1(config)# vsan 10 interface fc1/1
FC_switch_B_1(config)# vsan 10 interface fc1/13
FC_switch_B_1(config)# end
FC_switch_B_1# copy running-config startup-config

3. Verify the members of the VSAN using the following command:

show vsan member

Example

FC_switch_A_1# show vsan member

FC_switch_B_1# show vsan member

4. Configure the VSAN to guarantee in-order-delivery of frames:

78 | MetroCluster Installation and Configuration Guide

a. Enable the in-order-guarantee of exchanges for the VSAN by entering the following
command:
in-order-guarantee vsan vsan-ID

b. Enable load balancing for the VSAN by entering the following command:
vsan vsan-ID loadbalancing src-dst-id

Example
The following example shows the commands on FC_switch__A_1:

FC_switch_A_1# config t
FC_switch_A_1(config)# in-order-guarantee vsan 10
FC_switch_A_1(config)# vsan database
FC_switch_A_1(config-vsan-db)# vsan 10 loadbalancing src-dst-id
FC_switch_A_1(config-vsan-db)# end
FC_switch_A_1# copy running-config startup-config

The following example shows the commands on FC_switch__B_1:

FC_switch_B_1# config t
FC_switch_B_1(config)# in-order-guarantee vsan 10
FC_switch_B_1(config)# vsan database
FC_switch_B_1(config-vsan-db)# vsan 10 loadbalancing src-dst-id
FC_switch_B_1(config-vsan-db)# end
FC_switch_B_1# copy running-config startup-config

5. Set QoS policies for the FC-VI VSAN:

a. Enter configuration mode:

conf t

b. Enable the QoS and create a class map by entering the following commands in sequence:
qos enable
qos class-map class_name match-any

c. Add the class map created in a previous step to the policy map by entering the following
command:
class class_name

d. Set the priority by entering the following command:

priority high

e. Add the VSAN to the policy map created in step 2 by entering the following command:
qos service policy policy_name vsan vsanid

f. Copy the updated configuration to the startup configuration:

copy running-config startup-config

Example
The following example shows the commands on FC_switch__A_1:

FC_switch_A_1# conf t
FC_switch_A_1(config)# qos enable
FC_switch_A_1(config)# qos class-map FCVI_1_10_Class match-any
FC_switch_A_1(config)# qos policy-map FCVI_1_10_Policy
FC_switch_A_1(config-pmap)# class FCVI_1_10_Class
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 79

FC_switch_A_1(config-pmap-c)# priority high

FC_switch_A_1(config-pmap-c)# exit
FC_switch_A_1(config)# exit
FC_switch_A_1(config)# qos service policy FCVI_1_10_Policy vsan 10
FC_switch_A_1(config)# end
FC_switch_A_1# copy running-config startup-config

The following example shows the commands on FC_switch__B_1:

FC_switch_B_1# conf t
FC_switch_B_1(config)# qos enable
FC_switch_B_1(config)# qos class-map FCVI_1_10_Class match-any
FC_switch_B_1(config)# qos policy-map FCVI_1_10_Policy
FC_switch_B_1(config-pmap)# class FCVI_1_10_Class
FC_switch_B_1(config-pmap-c)# priority high
FC_switch_B_1(config-pmap-c)# exit
FC_switch_B_1(config)# exit
FC_switch_B_1(config)# qos service policy FCVI_1_10_Policy vsan 10
FC_switch_B_1(config)# end
FC_switch_B_1# copy running-config startup-config

6. Configure the storage VSAN:

a. Set the VSAN ID using the following command:
vsan vsan-ID

b. Set the VSAN name using the following command:

vsan vsan-ID name vsan_name

Example
The following example shows the commands on FC_switch_A_1:

FC_switch_A_1# conf t
FC_switch_A_1(config)# vsan database
FC_switch_A_1(config-vsan-db)# vsan 20
FC_switch_A_1(config-vsan-db)# vsan 20 name STOR_1_20
FC_switch_A_1(config-vsan-db)# end
FC_switch_A_1# copy running-config startup-config

The following example shows the commands on FC_switch_B_1:

FC_switch_B_1# conf t
FC_switch_B_1(config)# vsan database
FC_switch_B_1(config-vsan-db)# vsan 20
FC_switch_B_1(config-vsan-db)# vsan 20 name STOR_1_20
FC_switch_B_1(config-vsan-db)# end
FC_switch_B_1# copy running-config startup-config

7. Add ports to the storage VSAN.

For the storage VSAN, all ports connecting HBA or FC-to-SAS bridges must be added. In this
example fc1/5, fc1/9, fc1/17, fc1/21. fc1/25, fc1/29, fc1/33 and fc1/37 are being added.

Example
The following example shows the commands on FC_switch_A_1:

FC_switch_A_1# conf t
FC_switch_A_1(config)# vsan database
FC_switch_A_1(config)# vsan 20 interface fc1/5
FC_switch_A_1(config)# vsan 20 interface fc1/9
FC_switch_A_1(config)# vsan 20 interface fc1/17
80 | MetroCluster Installation and Configuration Guide

FC_switch_A_1(config)# vsan 20 interface fc1/21

FC_switch_A_1(config)# vsan 20 interface fc1/25
FC_switch_A_1(config)# vsan 20 interface fc1/29
FC_switch_A_1(config)# vsan 20 interface fc1/33
FC_switch_A_1(config)# vsan 20 interface fc1/37
FC_switch_A_1(config)# end
FC_switch_A_1# copy running-config startup-config

The following example shows the commands on FC_switch_B_1:

FC_switch_B_1# conf t
FC_switch_B_1(config)# vsan database
FC_switch_B_1(config)# vsan 20 interface fc1/5
FC_switch_B_1(config)# vsan 20 interface fc1/9
FC_switch_B_1(config)# vsan 20 interface fc1/17
FC_switch_B_1(config)# vsan 20 interface fc1/21
FC_switch_B_1(config)# vsan 20 interface fc1/25
FC_switch_B_1(config)# vsan 20 interface fc1/29
FC_switch_B_1(config)# vsan 20 interface fc1/33
FC_switch_B_1(config)# vsan 20 interface fc1/37
FC_switch_B_1(config)# end
FC_switch_B_1# copy running-config startup-config

Configuring the E-ports on the Cisco FC switch

You must configure the FC switch ports that connect the ISL. These are the E-ports, and
configuration must be done for each port. To do so, the correct number of buffer-to-buffer credits
(BBCs) must be calculated.

About this task

All ISLs in the fabric must be configured with the same speed and distance settings.
This task must be performed on each ISL port.

Steps

1. Use the following table to determine the adjusted required BBCs per kilometer for possible port
speeds.
To determine the correct number of BBCs, you multiply the Adjusted BBCs required
(determined from the table below) by the distance in kilometers between the
switches. The adjustment factor of 1.5 is required to account for FC-VI framing behavior.

Speed in Gbps BBCs required per Adjusted BBCs required (BBCs per
kilometer km x 1.5)
1 0.5 0.75
2 1 1.5
4 2 3
8 4 6
16 8 12

Example
For example, to compute the required number of credits for a distance of 30 km on a 4-Gbps link,
make the following calculation:

Speed in Gbps is 4

Adjusted BBCs required is 3.

 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 81

Distance in kilometers between switches is 30 km.

3 x 30 = 90

2. Issue the following command to enter configuration mode:

config t

3. Specify the port you are configuring by entering the following command:
interface port-name

4. Shut down the port:

shutdown

5. Set the rate mode of the port to dedicated:

switchport rate-mode dedicated

6. Set the speed for the port:

switchport speed speed

7. Set the buffer-to-buffer credits for the port:

switchport fcrxbbcredit number of buffers

8. Set the port to E mode:

switchport mode E

9. Enable the trunk mode for the port:

switchport trunk mode on

10. Add the ISL VSANs to the trunk:

switchport trunk allowed vsan 10
switchport trunk allowed vsan add 20

11. Add the port to port channel 1:

channel-group 1

12. Repeat the previous steps for the matching ISL port on the partner switch in the fabric.

Example
The following example shows port fc1/41 configured for a distance of 30 km and 8 Gbps:

FC_switch_A_1# conf t
FC_switch_A_1# shutdown
FC_switch_A_1# switchport rate-mode dedicated
FC_switch_A_1# switchport speed 8000
FC_switch_A_1# switchport fcrxbbcredit 60
FC_switch_A_1# switchport mode E
FC_switch_A_1# switchport trunk mode on
FC_switch_A_1# switchport trunk allowed vsan 10
FC_switch_A_1# switchport trunk allowed vsan add 20
FC_switch_A_1# channel-group 1
fc1/36 added to port-channel 1 and disabled

FC_switch_B_1# conf t
FC_switch_B_1# shutdown
FC_switch_B_1# switchport rate-mode dedicated
FC_switch_B_1# switchport speed 8000
FC_switch_B_1# switchport fcrxbbcredit 60
FC_switch_B_1# switchport mode E
FC_switch_B_1# switchport trunk mode on
82 | MetroCluster Installation and Configuration Guide

FC_switch_B_1# switchport trunk allowed vsan 10

FC_switch_B_1# switchport trunk allowed vsan add 20
FC_switch_B_1# channel-group 1
fc1/36 added to port-channel 1 and disabled

13. Issue the following command on both switches to restart the ports:
no shutdown

14. Repeat the previous steps for the other ISL ports in the fabric.

15. Add native vsan to port-channel interface on both switches in the same fabric:
interface port-channel number
switchport trunk allowed vsan add native_san_id

16. Verify configuration of the port-channel:

show interface port-channel number

The port channel should have the following attributes:

The port-channel is trunking.

Admin port mode is E, trunk mode is on.

Speed shows the cumulative value of all the ISL link speeds.
For example, two ISL ports operating at 4 Gbps should show a speed of 8 Gbps.

Trunk vsans (admin allowed and active) shows all the allowed VSANs.

Trunk vsans (up) shows all the allowed VSANs.

The member list shows all the ISL ports that were added to the port-channel.

The port VSAN number shuld be the same as the VSAN that contains the ISLs (usually native
vsan 1).

Example

FC_switch_A_1(config-if)# show int port-channel 1

port-channel 1 is trunking
Hardware is Fibre Channel
Port WWN is 24:01:54:7f:ee:e2:8d:a0
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Speed is 8 Gbps
Trunk vsans (admin allowed and active) (1,10,20)
Trunk vsans (up) (1,10,20)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
5 minutes input rate 1154832 bits/sec,144354 bytes/sec, 170
frames/sec
5 minutes output rate 1299152 bits/sec,162394 bytes/sec, 183
frames/sec
535724861 frames input,1069616011292 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
572290295 frames output,1144869385204 bytes
0 discards,0 errors
5 input OLS,11 LRR,2 NOS,0 loop inits
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 83

14 output OLS,5 LRR, 0 NOS, 0 loop inits

Member[1] : fc1/36
Member[2] : fc1/40
Interface last changed at Thu Oct 16 11:48:00 2014

17. Exit interface configuration on both switches:

end

18. Copy the updated configuration to the startup configuration on both fabrics:
copy running-config startup-config

Example

FC_switch_A_1(config-if)# end
FC_switch_A_1# copy running-config startup-config

FC_switch_B_1(config-if)# end
FC_switch_B_1# copy running-config startup-config

19. Repeat the previous steps on the second switch fabric.

Related concepts
Port assignments for FC switches in a four-node configuration on page 38

Configuring zoning on a Cisco FC switch

You must assign the switch ports to separate zones to isolate storage (HBA) and controller (FC-VI)
traffic.

About this task

These steps must be performed on both FC switch fabrics.

Steps

1. Clear the existing zones and zoneset if present.

a. Determine which zones and zonesets are active:

show zoneset active

Example

FC_switch_A_1# show zoneset active

FC_switch_B_1# show zoneset active

b. Disable the active zonesets identified in the previous step:

no zoneset activate name zoneset_name vsan vsan_id

Example
The following example shows two zonesets being disabled:

ZoneSet_A on FC_switch_A_1 in VSAN 10

ZoneSet_B on FC_switch_B_1 in VSAN 20

84 | MetroCluster Installation and Configuration Guide

FC_switch_A_1# no zoneset activate name ZoneSet_A vsan 10

FC_switch_B_1# no zoneset activate name ZoneSet_B vsan 20

c. After all zonesets are deactivated, clear the zone database:

clear zone database zone-name

Example

FC_switch_A_1# clear zone database 10

FC_switch_A_1# copy running-config startup-config

FC_switch_B_1# clear zone database 20

FC_switch_B_1# copy running-config startup-config

2. Obtain the switch worldwide name (WWN):

show wwn switch

3. Configure the basic zone settings:

a. Set the default zoning policy to deny:
no system default zone default-zone permit

b. Enable the full zone distribution:

system default zone distribute full

c. Set the default zoning policy for each VSAN:

no zone default-zone permit vsanid

d. Set the default full zone distribution for each VSAN:

zoneset distribute full vsanid

Example

FC_switch_A_1# conf t
FC_switch_A_1(config)# no system default zone default-zone permit
FC_switch_A_1(config)# system default zone distribute full
FC_switch_A_1(config)# no zone default-zone permit 10
FC_switch_A_1(config)# no zone default-zone permit 20
FC_switch_A_1(config)# zoneset distribute full vsan 10
FC_switch_A_1(config)# zoneset distribute full vsan 20
FC_switch_A_1(config)# end
FC_switch_A_1# copy running-config startup-config

FC_switch_B_1# conf t
FC_switch_B_1(config)# no system default zone default-zone permit
FC_switch_B_1(config)# system default zone distribute full
FC_switch_B_1(config)# no zone default-zone permit 10
FC_switch_B_1(config)# no zone default-zone permit 20
FC_switch_B_1(config)# zoneset distribute full vsan 10
FC_switch_B_1(config)# zoneset distribute full vsan 20
FC_switch_B_1(config)# end
FC_switch_B_1# copy running-config startup-config

4. Create storage zones and add the storage ports to them.

These steps only need to be performed on one switch in each fabric.
Each storage zone contains the HBA initiator ports from all controllers and one single port
connecting an FC-to-SAS bridge. Each zone has nine members.
 Cabling a four-node or two-node fabric-attached MetroCluster configuration | 85

a. Create the storage zones by entering the following command:

zone name STOR_zone-name vsan vsanid

b. Add storage ports to the zone by entering the following command:

member STOR_zone-name

c. Activate the zone set by entering the following command:

zoneset activate name STOR_zonenameesetname vsan vsanid

Example

FC_switch_A_1# conf t
FC_switch_A_1(config)# zone name STOR_Zone_1_20_25 vsan 20
FC_switch_A_1(config-zone)# member interface fc1/5 swwn
20:00:00:05:9b:24:cb:78
FC_switch_A_1(config-zone)# member interface fc1/9 swwn
20:00:00:05:9b:24:cb:78
FC_switch_A_1(config-zone)# member interface fc1/17 swwn
20:00:00:05:9b:24:cb:78
FC_switch_A_1(config-zone)# member interface fc1/21 swwn
20:00:00:05:9b:24:cb:78
FC_switch_A_1(config-zone)# member interface fc1/5 swwn
20:00:00:05:9b:24:12:99
FC_switch_A_1(config-zone)# member interface fc1/9 swwn
20:00:00:05:9b:24:12:99
FC_switch_A_1(config-zone)# member interface fc1/17 swwn
20:00:00:05:9b:24:12:99
FC_switch_A_1(config-zone)# member interface fc1/21 swwn
20:00:00:05:9b:24:12:99
FC_switch_A_1(config-zone)# member interface fc1/25 swwn
20:00:00:05:9b:24:cb:78
FC_switch_A_1(config-zone)# end
FC_switch_A_1# copy running-config startup-config

5. Create an FCVI zone set and add the FCVI ports to it:
These steps only need to be performed on one switch in the fabric.

a. Create the FCVI zone set by entering the following command:

zoneset name FCVI_zonesetname vsan vsanid

b. Add FCVI zones to the zone set by entering the following command:
member FCVI_zonename

c. Activate the zone set by entering the following command:

zoneset activate name FCVI_zonesetname vsan vsanid

Example

FC_switch_A_1# conf t
FC_switch_A_1(config)# zoneset name FCVI_Zoneset_1_20 vsan 20
FC_switch_A_1(config-zoneset)# member FCVI_Zone_1_20_25
FC_switch_A_1(config-zoneset)# member FCVI_Zone_1_20_29
...
FC_switch_A_1(config-zoneset)# exit
FC_switch_A_1(config)# zoneset activate name FCVI_ZoneSet_1_20 vsan 20
FC_switch_A_1(config)# exit
FC_switch_A_1# copy running-config startup-config

6. Verify the zoning:

show zone
86 | MetroCluster Installation and Configuration Guide

7. Repeat the previous steps on the second FC switch fabric.

Ensuring the FC switch configuration is saved

You must make sure the FC switch configuration is saved to the startup config on all switches.

Step

1. Issue the following command on both FC switch fabrics:

copy running-config startup-config

Example

FC_switch_A_1# copy running-config startup-config

FC_switch_B_1# copy running-config startup-config

 87

Cabling a two-node SAS-attached stretch

MetroCluster configuration
The MetroCluster components must be physically installed, cabled, and configured at both
geographic sites. The steps are slightly different for a system with native disk shelves as opposed to a
system with array LUNs.

About this task

Parts of a two-node direct-attached MetroCluster

configuration
The two-node MetroCluster direct-attached configuration requires a number of parts, including two
single-node clusters in which the controller modules afre directly connected to the storage using SAS
cables.
The MetroCluster configuration includes the following key hardware elements:

Storage controllers
The storage controllers connect directly to the storage using SAS cables.
Each storage controller is configured as a DR partner to a storage controller on the partner site.
When the MetroCluster is enabled, the system will automatically pair the two nodes with lowest
system IDs in each of the two clusters as DR partners.

Copper SAS cables can be used for shorter distances.

Optical SAS cables can be used for longer distances.

NetApp Interoperability Matrix Tool

After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

Cluster peering network

The cluster peering network provides connectivity for mirroring of the Storage Virtual Machine
(SVM) configuration. The configuration of all SVMs on one cluster is mirrored to the partner
cluster.
88 | MetroCluster Installation and Configuration Guide

Required MetroCluster components and naming guidelines

for two-node stretch configurations
The MetroCluster configuration requires a variety of hardware components. For convenience and
clarity, standard names for components are used throughout the MetroCluster documentation. Also,
one site is referred to as Site A and the other site is referred to as Site B.

Supported software and hardware

The hardware and software must be supported for the MetroCluster configuration.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray). You
use the Component Explorer to select the components and Data ONTAP version to refine your
search. You can click Show Results to display the list of supported configurations that match the
criteria.
NetApp Hardware Universe
When using All Flash Optimized systems, all controller modules in the MetroCluster configuration
must be configured as All Flash Optimized systems.

Required components
Because of the hardware redundancy in the MetroCluster configuration, there are two of each
component at each site. The sites are arbitrarily assigned the letters A and B and the individual
components are arbitrarily assigned the numbers 1 and 2.
The MetroCluster configuration also includes SAS storage shelves that connect to the FC-to-SAS
bridges.

Component Example names

Site A Site B
Two single-node Data ONTAP clusters cluster_A cluster_B
Naming must be unique within the MetroCluster
configuration.
Two storage controllers controller_A_1 controller_B_1
Naming must be unique within the MetroCluster
configuration.
All controller modules must be the same platform model
and running the same version of Data ONTAP.
The FC-VI connections between the controller modules
must be 4 Gbps.
 Cabling a two-node SAS-attached stretch MetroCluster configuration | 89

Component Example names

Site A Site B
Two FC-to-SAS bridges (ATTO 6500N FibreBridge bridge_A_1_port- bridge_B_1_po
bridges) per storage stack (required only if not using number rt-number
direct-attached SAS cabling)
bridge_A_2_port- bridge_B_2_po
These connect the SAS disk shelves to the controllers. number rt-number
Naming must be unique within the MetroCluster
configuration.
The suggested names used as examples in this guide
identify the controller that the bridge connects to and the
port.
The FC connections between the controller modules and
the FC-to-SAS bridges must be 4 Gbps.
At least eight SAS disk shelves (recommended) shelf_A_1_1 shelf_B_1_1
Four shelves are recommended at each site to allow disk shelf_A_2_1 shelf_B_2_1
ownership on a per-shelf basis. A minimum of two
shelves at each site is supported. shelf_B_1_2 shelf_A_1_2

Note: FlexArray systems support array LUNs and have shelf_B_2_2 shelf_A_2_2
different storage requirements.
Requirements for a MetroCluster configuration with
array LUNs on page 155

Installing and cabling MetroCluster components

SAS optical cables can be used to cable SAS disk shelves in a stretch MetroCluster system to achieve
greater distance connectivity.

Before you begin any procedure in this document

The following overall requirements must be met before completing this task:

If you are using SAS optical single-mode breakout cables, the following rules apply:

You can use these cables for controller-to-shelf connections.

Shelf-to-shelf connections use multimode QSFP-to-QSFP cables or multimode MPO cables
with MPO QSFP modules.

The point-to-point (QSFP-to-QSFP) path of a single single-mode cable cannot exceed 500
meters.

The total end-to-end path (sum of point-to-point paths from the controller to the last shelf)
cannot exceed 510 meters.
The total path includes the set of breakout cables, patch panels, and inter-panel cables.

Up to one pair of patch panels can be used in a path.

You need to supply the patch panels and inter-panel cables.

The inter-panel cables must be the same mode as the SAS optical breakout cable: single-
mode.

You must connect all eight (four pairs) of the SC, LC, or MTRJ breakout connectors to the
patch panel.
90 | MetroCluster Installation and Configuration Guide

The SAS cables can be SAS copper, SAS optical, or a mix depending on whether or not your
system meets the requirements for using the type of cable.
If you are using a mix of SAS copper cables and SAS optical cables, the following rules apply:

Shelf-to-shelf connections in a stack must be all SAS copper cables or all SAS optical cables.

If the shelf-to-shelf connections are SAS optical cables, the shelf-to-controller connections to
that stack must also be SAS optical cables.

If the shelf-to-shelf connections are SAS copper cables, the shelf-to-controller connections to
that stack can be SAS optical cables or SAS copper cables.
All components must be supported.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

About this task

The use of SAS optical cables in a stack attached to FibreBridge 6500N bridges is not supported.

Disk shelves connected with SAS optical cables require a version of disk shelf firmware that
supports SAS optical cables.
Best practice is to update all disk shelves in the storage system with the latest version of disk
shelf firmware.
Note: Do not revert disk shelf firmware to a version that does not support SAS optical cables.

The cable QSFP connector end connects to a disk shelf or a SAS port on a controller.
The QSFP connectors are keyed; when oriented correctly into a SAS port the QSFP connector
clicks into place and the disk shelf SAS port link LED, labeled LNK (Link Activity), illuminates
green. Do not force a connector into a port.

The terms node and controller are used interchangeably.

Choices
Racking the hardware components on page 90
Cabling the controllers to each other and the storage shelves on page 91
Cabling the cluster peering connections on page 92
Cabling the management and data connections on page 92

Racking the hardware components

If you have not received the equipment already installed in cabinets, you must rack the components.

About this task

This task must be performed on both MetroCluster sites.

Steps

1. Plan the positioning of the MetroCluster components.

The amount of rack space needed depends on the platform model of the storage controllers, the
switch types, and the number of disk shelf stacks in your configuration.
 Cabling a two-node SAS-attached stretch MetroCluster configuration | 91

2. Properly ground yourself.

3. Install the storage controllers in the rack or cabinet.

Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems
4. Install the disk shelves, power them on, and set the shelf IDs.
SAS Disk Shelves Installation and Service Guide for DS4243, DS2246, DS4486, and DS4246
You must power-cycle each disk shelf.

Shelf IDs must be unique for each SAS disk shelf within the entire MetroCluster configuration
(including both sites).

Cabling the controllers to each other and the storage shelves

The controller FC-VI adapters must be cabled directly to each other. The controller SAS ports must
be cabled to both the remote and local storage stacks.

About this task

This task must be performed on both MetroCluster sites.

Steps

1. Cable the FC-VI ports.

Controller A Controller B

fc-vi b fc-vi b

fc-vi a fc-vi a

2. Cable the SAS ports.

a. Determine the controller port pairs you will be cabling.

The Universal SAS and ACP Cabling Guide has completed SAS port pair worksheets for
common configurations, and also has a SAS port pairs worksheet template if a completed
worksheet is not provided for your configuration.
SAS Disk Shelves Universal SAS and ACP Cabling Guide
The following illustration shows the connections. Your port usage may be different, depending
on the available SAS ports on the controller module.
92 | MetroCluster Installation and Configuration Guide

Controller 1 Controller 2
Slot 1 Slot 1
A A
B B
C C
D D

Stack 1 Stack 2
ACP SAS ACP SAS

IOM A
First
shelf
IOM B

Last
shelf

Cabling the cluster peering connections

You must ensure that the controller ports used for cluster peering have connectivity with the cluster
on the partner site.

About this task

This task must be performed on each controller in the MetroCluster configuration.
At least two ports on each controller should be used for cluster peering.
The recommended minimum bandwidth for the ports and network connectivity is 1 GbE.

Step

1. Identify and cable at least two ports for cluster peering and ensure they have network connectivity
with the partner cluster.
Cluster peering can be done on dedicated ports or on data ports. Using dedicated ports ensures
higher throughput for the cluster peering traffic.

Related concepts
Considerations for configuring cluster peering on page 12

Related information
Clustered Data ONTAP 8.3 Data Protection Guide
Clustered Data ONTAP 8.3 Cluster Peering Express Guide

Cabling the management and data connections

You must cable the management and data ports on each storage controller to the site networks.

About this task

This task must be repeated for each controller at both MetroCluster sites.
You can connect the controller and cluster switch management ports to existing switches in your
network or to new dedicated network switches such as NetApp CN1601 cluster management
switches.
 Cabling a two-node SAS-attached stretch MetroCluster configuration | 93

Step

1. Cable the controller's management and data ports to the management and data networks at the
local site.
Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems
94

Cabling a two-node bridge-attached stretch

MetroCluster configuration
The MetroCluster components must be physically installed, cabled, and configured at both
geographic sites. The steps are slightly different for a system with native disk shelves as opposed to a
system with array LUNs.

About this task

Required MetroCluster components and naming guidelines

for two-node stretch configurations
The MetroCluster configuration requires a variety of hardware components. For convenience and
clarity, standard names for components are used throughout the MetroCluster documentation. Also,
one site is referred to as Site A and the other site is referred to as Site B.

Supported software and hardware

The hardware and software must be supported for the MetroCluster configuration.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray). You
use the Component Explorer to select the components and Data ONTAP version to refine your
search. You can click Show Results to display the list of supported configurations that match the
criteria.
NetApp Hardware Universe
When using All Flash Optimized systems, all controller modules in the MetroCluster configuration
must be configured as All Flash Optimized systems.
 Cabling a two-node bridge-attached stretch MetroCluster configuration | 95

Required components
Because of the hardware redundancy in the MetroCluster configuration, there are two of each
component at each site. The sites are arbitrarily assigned the letters A and B and the individual
components are arbitrarily assigned the numbers 1 and 2.
The MetroCluster configuration also includes SAS storage shelves that connect to the FC-to-SAS
bridges.

Component Example names

Site A Site B
Two single-node Data ONTAP clusters cluster_A cluster_B
Naming must be unique within the MetroCluster
configuration.
Two storage controllers controller_A_1 controller_B_1
Naming must be unique within the MetroCluster
configuration.
All controller modules must be the same platform model
and running the same version of Data ONTAP.
The FC-VI connections between the controller modules
must be 4 Gbps.
Two FC-to-SAS bridges (ATTO 6500N FibreBridge bridge_A_1_port- bridge_B_1_po
bridges) per storage stack (required only if not using number rt-number
direct-attached SAS cabling)
bridge_A_2_port- bridge_B_2_po
These connect the SAS disk shelves to the controllers. number rt-number
Naming must be unique within the MetroCluster
configuration.
The suggested names used as examples in this guide
identify the controller that the bridge connects to and the
port.
The FC connections between the controller modules and
the FC-to-SAS bridges must be 4 Gbps.
At least eight SAS disk shelves (recommended) shelf_A_1_1 shelf_B_1_1
Four shelves are recommended at each site to allow disk shelf_A_2_1 shelf_B_2_1
ownership on a per-shelf basis. A minimum of two
shelves at each site is supported. shelf_B_1_2 shelf_A_1_2

Note: FlexArray systems support array LUNs and have shelf_B_2_2 shelf_A_2_2
different storage requirements.
Requirements for a MetroCluster configuration with
array LUNs on page 155

Parts of a two-node MetroCluster configuration with FC-to-

SAS bridges
A two-node MetroCluster configuration consists of two single-node clusters and the networking
infrastructure between the clusters. The storage controllers are connected to the storage shelves with
FC-to-SAS bridges.
The MetroCluster configuration includes the following key hardware elements:

Storage controllers
96 | MetroCluster Installation and Configuration Guide

The storage controllers are not connected directly to the storage but connect to FC-to-SAS
bridges.
Each storage controller is configured as a DR partner to a storage controller on the partner site.
When the MetroCluster is enabled, the system will automatically pair the two nodes with lowest
system IDs in each of the two clusters as DR partners.
The storage controllers are connected to each other by FC cables between each controller's FC-VI
adapters.

FC-to-SAS bridges
The FC-to-SAS bridges connect the SAS storage stacks to the FC switches, providing bridging
between the two protocols.

Cluster peering network

The cluster peering network provides connectivity for mirroring of the Storage Virtual Machine
(SVM) configuration. The configuration of all SVMs on one cluster is mirrored to the partner
cluster.

The following illustration shows a simplified view of the MetroCluster configuration. For some
connections, a single line represents multiple, redundant connections between the components. Data
and management network connections are not shown.

Cluster peering network

controller_A_1 FC-VI links controller_B_1

FC_bridge_A_1 FC_bridge_B_1

SAS stack SAS stack

FC_bridge_A_2 FC_bridge_B_2

cluster_A cluster_B

The configuration consists of two single-node clusters.

Each site has one stack of SAS storage.

Additional storage stacks are supported, but only one is shown at each site.

Gathering required information and reviewing the workflow

You need to gather the required network information, identify the ports you will be using, and review
the hardware installation workflow before you begin cabling your system.

Related information
NetApp Interoperability Matrix Tool
 Cabling a two-node bridge-attached stretch MetroCluster configuration | 97

Worksheet for FC-to-SAS bridges

Before beginning to configure the MetroCluster sites, you should gather required configuration
information.

Site A, FC-to-SAS bridge 1 (FC_bridge_A_1_port-number)

Each SAS stack requires two FC-to-SAS bridges. One bridge connects to FC_switch_A_1_port-
number and the second connects to FC_switch_A_2_port-number.

Site A Your value

Bridge_A_1_port-number IP address
Bridge_A_1_port-number Username
Bridge_A_1_port-number Password

Site A, FC-to-SAS bridge 2 (FC_bridge_A_2_port-number)

Each SAS stack requires two FC-to-SAS bridges. One bridge connects to FC_switch_A_1_port-
number and the second connects to FC_switch_A_2_port-number.

Site A Your value

Bridge_A_2_port-number IP address
Bridge_A_2_port-number Username
Bridge_A_2_port-number Password

Site B, FC-to-SAS bridge 1 (FC_bridge_B_1_port-number)

Each SAS stack requires two FC-to-SAS bridges. One bridge connects to FC_switch_B_1_port-
number and the second connects to FC_switch_B_2_port-number.

Site B Your value

Bridge_B_1_port-number IP address
Bridge_B_1_port-number Username
Bridge_B_1_port-number Password

Site B, FC-to-SAS bridge 2 (FC_bridge_B_2_port-number)

Each SAS stack requires two FC-to-SAS bridges. One bridge connects to FC_switch_B_1_port-
number and the second connects to FC_switch_B_2_port-number.

Site B Your value

Bridge_B_2_port-number IP address
Bridge_B_2_port-number Username
Bridge_B_2_port-number Password
98 | MetroCluster Installation and Configuration Guide

Installing and cabling MetroCluster components

The storage controllers must be cabled to the FC switches and the ISLs must be cabled to link the
MetroCluster sites. The storage controllers must also be cabled to the data and management network.

Steps
1. Racking the hardware components on page 98
2. Cabling the controllers to each other on page 99
3. Cabling the cluster peering connections on page 99
4. Cabling the management and data connections on page 99

Racking the hardware components

If you have not received the equipment already installed in cabinets, you must rack the components.

About this task

This task must be performed on both MetroCluster sites.

Steps

1. Plan out the positioning of the MetroCluster components.

The rack space depends on the platform model of the storage controllers, the switch types, and the
number of disk shelf stacks in your configuration.

2. Properly ground yourself.

3. Install the storage controllers in the rack or cabinet.

Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems
4. Install the disk shelves, power them on, and set the shelf IDs.
SAS Disk Shelves Installation and Service Guide for DS4243, DS2246, DS4486, and DS4246
You must power-cycle each disk shelf.

Shelf IDs must be unique for each SAS disk shelf within the entire MetroCluster configuration
(including both sites).

5. Install each FC-to-SAS bridge:

a. Secure the L brackets on the front of the bridge to the front of the rack (flush-mount) with
the four screws.
The openings in the bridge L brackets are compliant with rack standard ETA-310-X for 19-
inch (482.6 mm) racks.
For more information and an illustration of the installation, see the ATTO FibreBridge 6500N
Installation and Operation Manual.
b. Connect each bridge to a power source that provides a proper ground.
 Cabling a two-node bridge-attached stretch MetroCluster configuration | 99

c. Power on each bridge.

Note: For maximum resiliency, you should ensure that bridges attached to the same stack of
disk shelves are connected to different power sources.

The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge
has completed its power-on self test sequence.

Cabling the controllers to each other

Each controller's FC-VI adapters must be cabled directly to its partner.

Step

1. Cable the FC-VI ports.

Controller A Controller B

fc-vi b fc-vi b

fc-vi a fc-vi a

Cabling the cluster peering connections

You must ensure that the controller ports used for cluster peering have connectivity with the cluster
on the partner site.

About this task

This task must be performed on each controller in the MetroCluster configuration.
At least two ports on each controller should be used for cluster peering.
The recommended minimum bandwidth for the ports and network connectivity is 1 GbE.

Step

1. Identify and cable at least two ports for cluster peering and ensure they have network connectivity
with the partner cluster.
Cluster peering can be done on dedicated ports or on data ports. Using dedicated ports ensures
higher throughput for the cluster peering traffic.

Related concepts
Considerations for configuring cluster peering on page 12

Related information
Clustered Data ONTAP 8.3 Data Protection Guide
Clustered Data ONTAP 8.3 Cluster Peering Express Guide

Cabling the management and data connections

You must cable the management and data ports on each storage controller to the site networks.

About this task

This task must be repeated for each controller at both MetroCluster sites.
100 | MetroCluster Installation and Configuration Guide

You can connect the controller and cluster switch management ports to existing switches in your
network or to new dedicated network switches such as NetApp CN1601 cluster management
switches.

Step

1. Cable the controller's management and data ports to the management and data networks at the
local site.
Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems

Installing FC-to-SAS bridges and SAS disk shelves

You install and cable ATTO FibreBridge FC-to-SAS bridges and SAS disk shelves as part of a new
MetroCluster installation.

About this task

For systems received from the factory, the FC-to-SAS bridges are preconfigured and do not require
additional configuration
This procedure is written with the assumption that you are using the recommended bridge
management interfaces: the ATTO ExpressNAV GUI and ATTO QuickNAV utility.
You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge
firmware. You use the ATTO QuickNAV utility to configure the bridge Ethernet management 1 port.
You can use other management interfaces instead, if needed, such as a serial port or Telnet to
configure and manage a bridge and to configure the Ethernet management 1 port, and FTP to update
the bridge firmware.
This procedure uses the following workflow:
 Cabling a two-node bridge-attached stretch MetroCluster configuration | 101

Steps
1. Preparing for the installation on page 101
2. Installing the FC-to-SAS bridge and SAS shelves in a fabric-attached configuration on page 102
3. Cabling the FC-to-SAS bridges to the controller module in a two-node direct-attached
configuration on page 106

Related concepts
Example of a four-node MetroCluster configuration with disks and array LUNs on page 194

Preparing for the installation

When you are preparing to install the FC-to-SAS bridges as part of your new MetroCluster system,
you must ensure that your system meets certain requirements, including meeting setup and
configuration requirements for the bridges. Other requirements include downloading the necessary
documents, the ATTO QuickNAV utility, and the bridge firmware.

Before you begin

Your system must already be installed in a rack if it was not shipped in a system cabinet.

Your configuration must be using supported hardware models and software versions.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

Each FC switch must have one FC port available for one bridge to connect to it.
102 | MetroCluster Installation and Configuration Guide

The computer you are using to set up the bridges must be running an ATTO-supported web
browser to use the ATTO ExpressNAV GUI.
The ATTO-supported web browsers are Internet Explorer 8 and 9, and Mozilla Firefox 3.
The ATTO Product Release Notes have an up-to-date list of supported web browsers. You can
access this document from the ATTO web site as described in the following steps.

Steps

1. Download the following documents:

SAS Disk Shelves Installation and Service Guide for DS4243, DS2246, DS4486, and DS4246
2. Download content from the ATTO web site and from the NetApp web site:

a. From NetApp Support, navigate to the ATTO FibreBridge Description page by clicking
Software, scrolling to Protocol Bridge and choosing ATTO FibreBridge from the drop-
down menu

b. Click Go!, and then clicking View & Download.

c. Access the ATTO web site using the link provided and download the following:

ATTO FibreBridge 6500N Installation and Operation Manual

ATTO QuickNAV utility (to the computer you are using for setup)

d. Navigate to the ATTO FibreBridge 6500N Firmware Download page by clicking Continue
at the end of the ATTO FibreBridge Description page.

Download the bridge firmware file using Steps 1 through 3 of that procedure.
You update the firmware on each bridge later in this procedure.

Make a copy of the ATTO FibreBridge 6500N Firmware Download page and release notes
for reference when you are instructed to update the firmware on each bridge.

3. Gather the hardware and information needed to use the recommended bridge management
interfaces, the ATTO ExpressNAV GUI, and the ATTO QuickNAV utility:

a. Acquire a standard Ethernet cable.

The cable connects from the bridge Ethernet management 1 port to your network.

b. Determine a non-default user name and password for accessing the bridges.
You should change the default user name and password.

c. Obtain an IP address, subnet mask, and gateway information for the Ethernet management 1
port on each bridge.

d. Disable VPN clients on the computer you are using for setup.
Active VPN clients cause the QuickNAV scan for bridges to fail.

Installing the FC-to-SAS bridge and SAS shelves in a fabric-attached

configuration
After ensuring that the system meets all the requirements in the Preparing for the installation
section, you can install your new system.

About this task

You should use an equal number of disk shelves at each site.

 Cabling a two-node bridge-attached stretch MetroCluster configuration | 103

The system connectivity requirements for maximum distances for disk shelves, FC switches, and
backup tape devices using 50-micron, multimode fiber-optic cables, also apply to FibreBridge
bridges.
The Site Requirements Guide has detailed information about system connectivity requirements.

Steps

1. Connect the Ethernet management 1 port on each bridge to your network using an Ethernet cable.
Note: The Ethernet management 1 port enables you to quickly download the bridge firmware
(using ATTO ExpressNAV or FTP management interfaces) and to retrieve core files and extract
logs.

2. Configure the Ethernet management 1 port for each bridge by following the procedure in the
ATTO FibreBridge 6500N Installation and Operation Manual, section 2.0.
Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet
management port that is connected by the Ethernet cable is configured. For example, if you
also wanted to configure the Ethernet management 2 port, you would need to connect the
Ethernet cable to port 2 and run QuickNAV.

3. Configure the bridges.

Be sure to make note of the user name and password that you designate.

a. Configure the bridge name.

The bridges should each have a unique name within the MetroCluster configuration.

b. Configure the data rate/speed of the bridge FC ports.

The most current information on supported distance can be found in the NetApp
Interoperability Matrix Tool.
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without
FlexArray). You use the Component Explorer to select the components and Data ONTAP
version to refine your search. You can click Show Results to display the list of supported
configurations that match the criteria.

c. Configure the connection mode that the bridges use to communicate across the FC network.
You must set the bridge connection mode to ptp (point-to-point).

For example, if you were to use the command line interface (CLI) to set the bridge FC 1 port's
basic required configuration, you would enter the following commands; the last command saves
the configuration changes:
set ipaddress
set subnet
set ipgateway
set FCDataRate 1 8Gb
set FCConnMode 1 ptp
set SNMP enabled
set bridgename
SaveConfiguration

Note: To set the IP address without the Quicknav utility, you need to have a serial connection
to the FibreBridge.

Result: You are prompted to restart the bridge.

104 | MetroCluster Installation and Configuration Guide

The ATTO FibreBridge 6500N Installation and Operation Manual has the most current
information on available commands and how to use them.

4. Update the firmware on each bridge to the latest version by following the instructionsstarting
with Step 4on the FibreBridge 6500N Download page.

5. Cable the disk shelves to the bridges:

a. Daisy-chain the disk shelves in each stack.

For information about daisy-chaining disk shelves, see the Installation and Service Guide for
your disk shelf model.
b. For each stack of disk shelves, cable IOM A square port of the first shelf to SAS port A on
FibreBridge A.

c. For each stack of disk shelves, cable IOM B circle port of the last shelf to SAS port A on
FibreBridge B.

Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack
through the first shelf, and bridge B connects to the B-side of the stack through the last shelf.
Note: The bridge SAS port B is disabled.

The following illustration shows a set of bridges cabled to a stack of three disk shelves:

FC1 FC2 FC1 FC2

FC_bridge_x_1_06 FC_bridge_x_2_o6

M1 SAS A M1 SAS A

Stack of SAS shelves

IOM A IOM B
First
shelf

Last
shelf

6. Verify that each bridge can detect all disk drives and disk shelves it is connected to.
 Cabling a two-node bridge-attached stretch MetroCluster configuration | 105

If you are using the... Then...

ATTO ExpressNAV GUI
a. In a supported web browser, enter the IP address of a bridge in the
browser box.
You are brought to the ATTO FibreBridge 6500N home page, which
has a link.

b. Click the link, and then enter your user name and the password that
you designated when you configured the bridge.
The ATTO FibreBridge 6500N status page appears with a menu to
the left.

c. Click Advanced in the menu.

d. Enter the following command, and then click Submit:

sastargets

Serial port connection Enter the following command:

sastargets

Example
The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines
are sequentially numbered so that you can quickly count the devices. For example, the following
output shows that 10 disks are connected:

Tgt VendorID ProductID Type SerialNumber

0 NETAPP X410_S15K6288A15 DISK 3QP1CLE300009940UHJV
1 NETAPP X410_S15K6288A15 DISK 3QP1ELF600009940V1BV
2 NETAPP X410_S15K6288A15 DISK 3QP1G3EW00009940U2M0
3 NETAPP X410_S15K6288A15 DISK 3QP1EWMP00009940U1X5
4 NETAPP X410_S15K6288A15 DISK 3QP1FZLE00009940G8YU
5 NETAPP X410_S15K6288A15 DISK 3QP1FZLF00009940TZKZ
6 NETAPP X410_S15K6288A15 DISK 3QP1CEB400009939MGXL
7 NETAPP X410_S15K6288A15 DISK 3QP1G7A900009939FNTT
8 NETAPP X410_S15K6288A15 DISK 3QP1FY0T00009940G8PA
9 NETAPP X410_S15K6288A15 DISK 3QP1FXW600009940VERQ

Note: If the text response truncated appears at the beginning of the output, you can use
Telnet to connect to the bridge and enter the same command to see all of the output.

7. Verify that the command output shows that the bridge is connected to all disks and disk shelves in
the stack that it is supposed to be connected to.

If the output is... Then...

Correct Repeat step 6 on page 47 for each remaining bridge.
Not correct
a. Check for loose SAS cables or correct the SAS cabling by repeating
step 5 on page 47.

b. Repeat step 6 on page 47.

106 | MetroCluster Installation and Configuration Guide

Cabling the FC-to-SAS bridges to the controller module in a two-node

direct-attached configuration
You must cable the bridges to the controller module.

Steps

1. Cable each bridge to the controllers:

a. Cable FC port 1 of the bridge to an 8-Gb or 4-Gb FC port on the controller in cluster_A.
b. Cable FC port 2 of bridge to the same speed FC port on the controller at cluster_A.

2. Repeat the previous step on the other bridges until all bridges have been cabled.
 107

Configuring hardware for sharing a Brocade 6510

FC fabric during transition
If your 7-Mode fabric MetroCluster configuration uses Brocade 6510 switches, you can share the
existing switch fabrics with the new clustered MetroCluster configuration. Shared switch fabrics
means the new MetroCluster configuration does not require a new, separate switch fabric. This
temporary configuration is only supported with the Brocade 6510 switch for transition purposes.

Before you begin

The 7-Mode fabric MetroCluster must be using Brocade 6510 switches.

If the MetroCluster configuration is currently not using Brocade 6510 switches, the switches must
be upgraded to Brocade 6510 prior to using this procedure.

The 7-Mode fabric MetroCluster configuration must be using SAS storage shelves only.
If the existing configuration includes FC storage shelves (such as the DS14mk4 FC), FC switch
fabric sharing is not supported.

All 48 ports of the Brocade 6510 switches must be licensed.

The SFPs on the switch ports used by the new, clustered MetroCluster configuration must support
16-Gbps rates.
The existing 7-Mode fabric MetroCluster can remain connected to ports using 8-Gbps or 16-Gbps
SFPs.

On each of the four Brocade 6510 switches, ports 24 through 45 must be available to connect the
ports of the new MetroCluster components.

You should ensure that the existing Inter-Switch Links (ISLs) are on ports 46 and 47.

The Brocade 6510 switches must be running a FOS firmware version that is supported on both
the 7-Mode fabric MetroCluster and clustered Data ONTAP MetroCluster configuration.

After you finish

After sharing the fabric and completing the MetroCluster configuration, you can transition data from
the 7-Mode fabric MetroCluster configuration.
After transitioning the data, you can remove the 7-Mode fabric MetroCluster cabling and, if desired,
move the clustered Data ONTAP MetroCluster cabling to the lower-numbered ports previously used
for the 7-Mode MetroCluster cabling. The ports are shown in Reviewing FC switch port assignments
for a four node MetroCluster on page 38. You must adjust the zoning for the rearranged ports.

Steps
1. Reviewing Brocade license requirements on page 108
2. Racking the hardware components on page 108
3. Cabling the new MetroCluster controllers to the existing FC fabrics on page 109
4. Configuring switch fabrics sharing between the 7-Mode and clustered MetroCluster configuration
on page 110

Related information
7-Mode Transition Tool 2.1 Data and Configuration Transition Guide
108 | MetroCluster Installation and Configuration Guide

Reviewing Brocade license requirements

You need certain licenses for the switches in a MetroCluster configuration. You must install these
licenses on all four switches.
The MetroCluster configuration has the following Brocade license requirements:

Trunking license for systems using more than one ISL, as recommended.

Extended Fabric license (for ISL distances over 6 km)

Enterprise license for sites with more than one ISL and an ISL distance greater than 6 km
The Enterprise license includes Brocade Network Advisor and all licenses except for additional
port licenses.

You can verify that the licenses are installed by using the licenseshow command. If you do not
have these licenses, contact your sales representative before proceeding.

Racking the hardware components

If you have not received the equipment already installed in cabinets, you must rack the components.

About this task

This task must be performed on both MetroCluster sites.

Steps

1. Plan out the positioning of the MetroCluster components.

The rack space depends on the platform model of the storage controllers, the switch types, and the
number of disk shelf stacks in your configuration.

2. Properly ground yourself.

3. Install the storage controllers in the rack or cabinet.

Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems
4. Install the FC switches in the rack or cabinet.

5. Install the disk shelves, power them on, and set the shelf IDs.
SAS Disk Shelves Installation and Service Guide for DS4243, DS2246, DS4486, and DS4246
You must power-cycle each disk shelf.

Shelf IDs must be unique for each SAS disk shelf within the entire MetroCluster configuration
(including both sites).

6. Install each FC-to-SAS bridge:

a. Secure the L brackets on the front of the bridge to the front of the rack (flush-mount) with
the four screws.
 Configuring hardware for sharing a Brocade 6510 FC fabric during transition | 109

The openings in the bridge L brackets are compliant with rack standard ETA-310-X for 19-
inch (482.6 mm) racks.
For more information and an illustration of the installation, see the ATTO FibreBridge 6500N
Installation and Operation Manual.
b. Connect each bridge to a power source that provides a proper ground.

c. Power on each bridge.

Note: For maximum resiliency, ensure that bridges attached to the same stack of disk
shelves are connected to different power sources.

The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge
has completed its power-on self test sequence.

Cabling the new MetroCluster controllers to the existing FC

fabrics
On each controller module in the clustered Data ONTAP MetroCluster configuration, the FC-VI
adapter and HBAs must be cabled to specific ports on the existing FC switches.

Steps

1. Cable the FC-VI and HBA ports according to the following table:

Site A Site B
Connect this Site A FC_switch_ Connect this Site B FC_switch_B_1
component and port... A_1 port... component and port... port...
controller_A_1 FC-VI port 1 32 controller_B_1 FC-VI port 32
1
controller_A_1 HBA port 1 33 controller_B_1 HBA port 33
1
controller_A_1 HBA port 2 34 controller_B_1 HBA port 34
2
controller_A_2 FC-VI port 1 35 controller_B_2 FC-VI port 35
1
controller_A_2 HBA 1 36 controller_B_2 HBA 1 36
controller_A_2 HBA 2 37 controller_B_2 HBA 2 37

2. Cable each FC-SAS bridge in the first switch fabric to the FC switches.
The number of bridges varies depending on the number of SAS storage stacks.

Site A Site B
Cable this site A bridge... FC_switch Cable this Site B bridge... FC_switch_B_1
_A_1 port...
port...
bridge_A_1_38 38 bridge_B_1_38 38
bridge_A_1_39 39 bridge_B_1_39 39
bridge_A_1_40 40 bridge_B_1_40 40
110 | MetroCluster Installation and Configuration Guide

Site A Site B
Cable this site A bridge... FC_switch Cable this Site B bridge... FC_switch_B_1
_A_1 port...
port...
bridge_A_1_41 41 bridge_B_1_41 41
bridge_A_1_42 42 bridge_B_1_42 42
bridge_A_1_43 43 bridge_B_1_43 43
bridge_A_1_44 44 bridge_B_1_44 44
bridge_A_1_45 45 bridge_B_1_45 45

3. Cable each bridge in the second switch fabric to the FC switches.

The number of bridges varies depending on the number of SAS storage stacks.

Site A Site B
Cable this site A bridge... FC_switch Cable this Site B bridge... FC_switch_B_2
_A_2 port...
port...
bridge_A_2_38 38 bridge_B_2_38 38
bridge_A_2_39 39 bridge_B_2_39 39
bridge_A_2_40 40 bridge_B_2_40 40
bridge_A_2_41 41 bridge_B_2_41 41
bridge_A_2_42 42 bridge_B_2_42 42
bridge_A_2_43 43 bridge_B_2_43 43
bridge_A_2_44 44 bridge_B_2_44 44
bridge_A_2_45 45 bridge_B_2_45 45

Configuring switch fabrics sharing between the 7-Mode and

clustered MetroCluster configuration
To share switch fabrics between the existing 7-Mode fabric MetroCluster and the new MetroCluster
configuration, you must set up specific zoning and other settings that are different than an unshared
configuration.

About this task

This task must be performed on both switch fabrics, one at a time.

Disabling one of the switch fabrics

You must disable one of the switch fabrics so you can modify its configuration. After you complete
the configuration and reenable the switch fabric, you will repeat the process on the other fabric.

Before you begin

You must have run the fmc_dc utility on the existing 7-Mode fabric MetroCluster configuration and
resolved any issues prior to beginning the configuration process.
 Configuring hardware for sharing a Brocade 6510 FC fabric during transition | 111

About this task

To ensure continued operation of the MetroCluster configuration, you must not disable the second
fabric while the first fabric is disabled.

Steps

1. Disable each of the switches in the fabric:

switchCfgPersistentDisable

If this command is not available, use the switchDisable command.

Example
The following example shows the command issued on FC_switch_A_1:

FC_switch_A_1:admin> switchCfgPersistentDisable

The following example shows the command issued on FC_switch_B_1:

FC_switch_B_1:admin> switchCfgPersistentDisable

2. Ensure that the 7-Mode MetroCluster configuration is functioning correctly using the redundant
fabric:

a. Confirm that controller failover is healthy:

cf status

Example

node_A> cf status
Controller Failover enabled, node_A is up.
VIA Interconnect is up (link 0 down, link 1 up).

b. Confirm that disks are visible:

storage show disk p

Example

node_A> storage show disk p

PRIMARY PORT SECONDARY PORT SHELF BAY

--------------------------- ---- ------------------ ---- ---------
Brocade-6510-2K0GG:5.126L27 B 1 0
Brocade-6510-2K0GG:5.126L28 B 1 1
Brocade-6510-2K0GG:5.126L29 B 1 2
Brocade-6510-2K0GG:5.126L30 B 1 3
Brocade-6510-2K0GG:5.126L31 B 1 4

.
.
.

c. Confirm that the aggregates are healthy:

aggr status
112 | MetroCluster Installation and Configuration Guide

Example

node_A> aggr status

Aggr State Status Options
aggr0 online raid_dp, aggr root, nosnap=on
mirrored
64-bit

Deleting TI zoning and configuring IOD settings

You must delete the existing TI zoning and reconfigure in-order-delivery (IOD) settings on the switch
fabric.

Steps

1. Identify the TI zones that are configured on the fabric:

zone -show

Example
The following example shows the zone FCVI_TI_FAB_2.

Brocade-6510:admin> zone --show

Defined TI zone configuration:
TI Zone Name: FCVI_TI_FAB_2
Port List: 1,0; 1,3; 2,0; 2,3
configured Status: Activated / Failover-Disabled
Enabled Status: Activated / Failover-Disabled

2. Delete the TI zones:

zone -delete zone-name

Example
The following example shows the deletion of zone FCVI_TI_FAB_2.

Brocade-6510:admin> zone --delete FCVI_TI_FAB_2

3. Confirm that the zones have been deleted:

zone -show

Example
The output should be similar to the following:

Brocade-6510:admin> zone -show

Defined TI zone configuration:

no TI zone configuration defined

4. Save the configuration:

cfgsave

5. Enable in-order-delivery:
iodset
 Configuring hardware for sharing a Brocade 6510 FC fabric during transition | 113

6. Select Advanced Performance Tuning (APT) policy 1, the Port Based Routing Policy:
aptpolicy 1

7. Disable Dynamic Load Sharing (DLS):

dlsreset

8. Verify the IOD settings using the followign commands:

iodshow
aptpolicy
dlsshow

Example
The output should be similar to the following:

Brocade-6510:admin> iodshow

IOD is set

Brocade-6510:admin> aptpolicy
Current Policy: 1

3 : Default Policy
1: Port Based Routing Policy
2: Device Based Routing Policy (FICON support only)
3: Exchange Based Routing Policy
Brocade-6510:admin> dlsshow

DLS is not set

Ensuring ISLs are in the same port group and configuring zoning
You must make sure that the Inter-Switch Links (ISLs) are in the same port group and configure
zoning for the MetroCluster configurations to successfully share the switch fabrics.

Steps

1. If the ISLs are not in the same port group, move one of the ISL ports to the same port group as the
other one.
You can use any available port except 32 through 45, which are used by the new MetroCluster
configuration. The recommended ISL ports are 46 and 47.

2. Follow the steps in Configuring zoning on a Brocade FC switch on page 62 to enable trunking
and the QoS zone.
The port numbers when sharing fabrics are different than those shown in the section. When
sharing, use ports 46 and 47 for the ISL ports. If you moved your ISL ports, you need to use the
procedure in Configuring the E-ports (ISL ports) on a Brocade FC switch on page 57 to configure
the ports.

3. Follow the steps in Configuring the non-E ports on the Brocade switch on page 61 to configure
the non-E ports.

4. Do not delete the zones or zone sets that already exist in the backend switches (for the 7-Mode
fabric MetroCluster) except the Traffic Isolation (TI) zones in Step 3.

5. Follow the steps in Configuring the E-ports (ISL ports) on a Brocade FC switch on page 57 to add
the zones required by the new MetroCluster to the existing zone sets.
114 | MetroCluster Installation and Configuration Guide

Example
The following example shows the commands and system output for creating the zones:

Brocade-6510-2K0GG:admin> zonecreate "QOSH2_FCVI_1", "2,32; 2,35;

1,32; 1,35"

Brocade-6510-2K0GG:admin> zonecreate "STOR_A_2_47", "2,33; 2,34;

2,36; 2,37; 1,33; 1,34; 1,36; 1,37; 1,47"

Brocade-6510-2K0GG:admin> zonecreate "STOR_B_2_47", "2,33; 2,34;

2,36; 2,37; 1,33; 1,34; 1,36; 1,37; 2,47"

Brocade-6510-2K0GG:admin> cfgadd config_1_FAB2, "QOSH2_FCVI_1;

STOR_A_2_47; STOR_B_2_47"

Brocade-6510-2K0GG:admin> cfgenable "config_1_FAB2"

You are about to enable a new zoning configuration.
This action will replace the old zoning configuration with the
current configuration selected. If the update includes changes
to one or more traffic isolation zones, the update may result in
localized disruption to traffic on ports associated with
the traffic isolation zone changes
Do you want to enable 'config_1_FAB2' configuration (yes, y, no, n):
[no] yes

Brocade-6510-2K0GG:admin> cfgsave
You are about to save the Defined zoning configuration. This
action will only save the changes on Defined configuration.
Do you want to save the Defined zoning configuration only? (yes, y,
no, n): [no] yes
Nothing changed: nothing to save, returning ...
Brocade-6510-2K0GG:admin>

Reenabling the switch fabric and verify operation

You must enable the FC switch fabric and make sure that the switches and devices are operating
correctly.

Steps

1. Enable the switches:

switchCfgPersistentEnable

If this command is not available, the switch should be in the

enabled
state after the fastBoot command is issued.

Example
The following example shows the command issued on FC_switch_A_1:

FC_switch_A_1:admin> switchCfgPersistentEnable

The following example shows the command issued on FC_switch_B_1:

FC_switch_B_1:admin> switchCfgPersistentEnable

2. Verify that the switches are online and all devices are properly logged in:
switchShow
 Configuring hardware for sharing a Brocade 6510 FC fabric during transition | 115

Example
The following example shows the command issued on FC_switch_A_1:

FC_switch_A_1:admin> switchShow

The following example shows the command issued on FC_switch_B_1:

FC_switch_B_1:admin> switchShow

3. Run the fmc_dc utility to ensure that the 7-Mode fabric MetroCluster is functioning correctly.
You can ignore errors related to Traffic Isolation (TI) zoning and trunking.

4. Repeat the tasks for the second switch fabric.

116

Configuring the MetroCluster software in Data

ONTAP
You must set up each node in the MetroCluster in Data ONTAP, including the node-level
configurations and the configuration of the nodes into two sites. Finally, you implement the
MetroCluster relationship between the two sites. The steps for systems with native disk shelves are
slightly different than those for systems with array LUNs.
For new systems configured in the factory, you do not need to configure the Data ONTAP software
except to change the preconfigured IP addresses. If your system is new, you can proceed with
verifying the configuration.

Steps
1. Gathering required information and reviewing the workflow on page 117
2. Similarities and differences between regular cluster and MetroCluster configurations on page 122
3. Setting a previously used controller module to system defaults in Maintenance mode on page 122
4. Verifying disk assignment in Maintenance mode in a four-node configuration on page 123
5. Verifying disk assignment in Maintenance mode in a two-node configuration on page 128
6. Verifying the HA state of components is mcc or mcc-2n in Maintenance mode on page 129
7. Setting up Data ONTAP on page 130
8. Configuring the clusters into a MetroCluster configuration on page 135
9. Checking for MetroCluster configuration errors with Config Advisor on page 151
 Configuring the MetroCluster software in Data ONTAP | 117

10. Verifying local HA operation on page 151

11. Verifying switchover, healing, and switchback on page 153
12. Installing the MetroCluster Tiebreaker software on page 153
13. Protecting configuration backup files on page 153

Gathering required information and reviewing the workflow

You need to gather the required IP addresses for the controller and review the software installation
workflow before you begin the configuration process.

IP network information worksheet for site A

You must obtain IP addresses and other network information for the first MetroCluster site (site A)
from your network administrator before you configure the system.

Site A switch information (if not using two-node switchless cluster configuration or
two-node MetroCluster configuration)
When you cable the system, you need a host name and management IP address for each cluster
switch. This information if using two-node switchless cluster or using two-node MetroCluster
configuration (one node at each site).

Cluster switch Host name IP address Network mask Default gateway

Interconnect 1
Interconnect 2
Management 1
Management 2

Site A cluster creation information

When you first create the cluster, you need the following information:

Type of information Your values

Cluster name
Example used in this guide:
site_A
DNS domain
DNS name servers
Location
Administrator password

Site A node information

For each node in the cluster, you need a management IP address, a network mask, and a default
gateway:
118 | MetroCluster Installation and Configuration Guide

Node Port IP address Network mask Default gateway

Node 1
Example used
in this guide:
controller_A_1
Node 2
Not required if
using two-node
MetroCluster
configuration
(one node at
each site).
Example used
in this guide:
controller_A_2

Site A LIFs and ports for cluster peering

For each node in the cluster, you need the IP addresses of two intercluster LIFs including a network
mask and a default gateway. The intercluster LIFs are used to peer the clusters.

Node Port IP address of Network mask Default gateway

intercluster LIF
Node 1 IC LIF
1
Node 1 IC LIF
2
Node 2 IC LIF
1
Not required if
using two-node
MetroCluster
configuration
(one node at
each site).
Node 2 IC LIF
2
Not required if
using two-node
MetroCluster
configuration
(one node at
each site).

Site A time server information

You must synchronize the time, which will require one or more NTP time servers:

Node Host name IP address Network mask Default gateway

NTP server 1
NTP server 2
 Configuring the MetroCluster software in Data ONTAP | 119

Site A AutoSupport information

You must configure AutoSupport on each node, which will require the following information:

Type of information Your values

From email address
Mail hosts IP addresses or names
Transport protocol HTTP, HTTPS, or
SMTP
Proxy server
Recipient email Full-length messages
addresses or
Concise messages
distribution lists
Partners

Site A service processor information

You must enable access to the service processor of each node for troubleshooting and maintenance,
which requires the following network information for each node:

Node IP address Network mask Default gateway

Node 1
Node 2
Not required if
using two-node
MetroCluster
configuration
(one node at each
site).

IP network information worksheet for site B

You must obtain IP addresses and other network information for the second MetroCluster site (site B)
from your network administrator before you configure the system.

Site B switch information (if not using two-node switchless cluster configuration or
two-node MetroCluster configuration)
When you cable the system, you need a host name and management IP address for each cluster
switch. This information if using two-node switchless cluster or using two-node MetroCluster
configuration (one node at each site).

Cluster switch Host name IP address Network mask Default gateway

Interconnect 1
Interconnect 2
Management 1
Management 2

Site B cluster creation information

When you first create the cluster, you need the following information:
120 | MetroCluster Installation and Configuration Guide

Type of information Your values

Cluster name
Example used in this guide:
site_B
DNS domain
DNS name servers
Location
Administrator password

Site B node information

For each node in the cluster, you need a management IP address, a network mask, and a default
gateway:

Node Port IP address Network mask Default gateway

Node 1
Example used
in this guide:
controller_B_1
Node 2
Not required if
using two-node
MetroCluster
configuration
(one node at
each site).
Example used
in this guide:
controller_B_2

Site B LIFs and ports for cluster peering

For each node in the cluster, you need the IP addresses of two intercluster LIFs including a network
mask and a default gateway. The intercluster LIFs are used to peer the clusters.

Node Port IP address of Network mask Default gateway

intercluster LIF
Node 1 IC LIF
1
Node 1 IC LIF
2
Node 2 IC LIF
1
Not required if
using two-node
MetroCluster
configuration
(one node at
each site).
 Configuring the MetroCluster software in Data ONTAP | 121

Node Port IP address of Network mask Default gateway

intercluster LIF
Node 2 IC LIF
2
Not required if
using two-node
MetroCluster
configuration
(one node at
each site).

Site B time server information

You must synchronize the time, which requires one or more NTP time servers:

Node Host name IP address Network mask Default gateway

NTP server 1
NTP server 2

Site B AutoSupport information

You must configure AutoSupport on each node, which requires the following information:

Type of information Your values

From email address
Mail hosts IP addresses or names
Transport protocol HTTP, HTTPS, or
SMTP
Proxy server
Recipient email Full-length messages
addresses or
Concise messages
distribution lists
Partners

Site B service processor information

You must enable access to the service processor of each node for troubleshooting and maintenance,
which requires the following network information for each node:

Node IP address Network mask Default gateway

Node 1
(controller_B_1)
Node 2
(controller_B_2)
Not required if
using two-node
MetroCluster
configuration
(one node at each
site).
122 | MetroCluster Installation and Configuration Guide

Similarities and differences between regular cluster and

MetroCluster configurations
The configuration of the nodes in each cluster in a MetroCluster configuration is similar to that of
nodes in a standard cluster.
The MetroCluster configuration is built on two standard clusters. Physically, the configuration must
be symmetrical, with each node having the same hardware configuration, and all the MetroCluster
components must be cabled and configured. But the basic software configuration for nodes in a
MetroCluster configuration is the same as that for nodes in a standard cluster.

Configuration step Standard cluster MetroCluster

configuration configuration
Configure management, cluster, and data LIFs Same in both types of clusters
on each node
Configure root aggregate
Configure nodes in the cluster as HA pairs
Set up cluster on one node in the cluster
Join the other node to the cluster
Create a mirrored root aggregate Optional Required
Create a mirrored data aggregate on each node Optional Required
Peer the clusters Optional Required
Enable the MetroCluster configuration Does not apply Required

Setting a previously used controller module to system

defaults in Maintenance mode
If your controller moduleshave been used previously, you must reset them to ensure a successful
MetroCluster configuration.

About this task

Important: This task is required only on controller modules that have been previously configured;
you do not need to perform this task if you received the controller modules as part of a new
MetroCluster configuration.

Steps

1. In Maintenance mode, return the environmental variables to their default setting:

set-defaults

2. Configure the settings for any HBA adapters in the system:

If you have this type of Use this command for configuration...

adapter and desired mode...
CNA FC ucadmin modify -mode fc -type initator
adapter_name
CNA Ethernet ucadmin modify -mode cna adapter_name
 Configuring the MetroCluster software in Data ONTAP | 123

If you have this type of Use this command for configuration...

adapter and desired mode...
FC target fcadmin config -t target adapter_name
FC initiator fcadmin config -t initiator adapter_name

3. Exit Maintenance mode:

halt

After you issue the command, wait until the system stops at the LOADER prompt.

4. Boot the node back into Maintenance mode to enable the configuration changes to take effect.

5. Verify the values of the variables:

If you have this type of Use this command...

adapter...
CNA ucadmin show
FC fcadmin show

6. Clear the system configuration:

wipeconfig

Verifying disk assignment in Maintenance mode in a four-

node configuration
Before fully booting the system to Data ONTAP, you can optionally boot to Maintenance mode and
verify the disk assignment on the nodes. The disks should be assigned to create a fully symmetric
active-active configuration, where each node and each pool have an equal number of disks assigned
to them.

About this task

New MetroCluster systems have disk assignment completed prior to shipment.
The following table shows example pool assignments for a MetroCluster configuration. Disks are
assigned to pools on a per-shelf basis.
124 | MetroCluster Installation and Configuration Guide

Disk shelf (example At site... Belongs to... And is assigned to

name)... that node's...

Disk shelf 1 Site A Node A 1 Pool 0

(shelf_A_1_1)
Disk shelf 2
(shelf_A_1_3)
Disk shelf 3 Node B 1 Pool 1
(shelf_B_1_1)
Disk shelf 4
(shelf_B_1_3)
Disk shelf 5 Node A 2 Pool 0
(shelf_A_2_1)
Disk shelf 6
(shelf_A_2_3)
Disk shelf 7 Node B 2 Pool 1
(shelf_B_2_1)
Disk shelf 8
(shelf_B_2_3)
Disk shelf 9 Site B Node B 1 Pool 0
(shelf_B_1_2)
Disk shelf 10
(shelf_B_1_4)
Disk shelf 11 Node A 1 Pool 1
(shelf_A_1_2)
Disk shelf 12
(shelf_A_1_4)
Disk shelf 13 Node B 2 Pool 0
(shelf_B_2_2)
Disk shelf 14
(shelf_B_2_4)
Disk shelf 15 Node A 2 Pool 1
(shelf_A_2_2)
Disk shelf 16
(shelf_A_2_4)

Steps

1. Confirm the shelf assignments:

disk show v

2. If necessary, explicitly assign disks on the attached disk shelves to the appropriate pool with the
disk assign command.

Using wildcards in the command enables you to assign all the disks on a disk shelf with one
command.

3. Show the disk shelf IDs and bays for each disk:
storage show disk x
 Configuring the MetroCluster software in Data ONTAP | 125

Assigning disk ownership in non-AFF systems

If the MetroCluster nodes do not have the disks correctly assigned, you must assign disks to each of
the nodes in the MetroCluster configuration on a shelf-by-shelf basis. You will create a configuration
in which each node has the same number of disks in its local and remote disk pools.

Before you begin

The storage controllers must be in Maintenance mode.

About this task

This task is not required if disks were correctly assigned when received from the factory.
Note: Pool 0 always contains the disks that are found at the same site as the storage system that
owns them.
Pool 1 always contains the disks that are remote to the storage system that owns them.

Steps

1. If you have not done so, boot each system into Maintenance mode.

2. Assign the disk shelves to the nodes located at the first site (site A):
Disk shelves at the same site as the node are assigned to pool 0 and disk shelves located at the
partner site are assigned to pool 1.
You should assign an equal number of shelves to each pool.

a. On the first node, systematically assign the local disk shelves to pool 0 and the remote disk
shelves to pool 1:
disk assign -shelf local-switch-name:shelf-name.port -p pool

Example
If storage controller Controller_A_1 has four shelves, you issue the following commands:

*> disk assign -shelf FC_switch_A_1:1-4.shelf1 -p 0

*> disk assign -shelf FC_switch_A_1:1-4.shelf2 -p 0

*> disk assign -shelf FC_switch_B_1:1-4.shelf1 -p 1

*> disk assign -shelf FC_switch_B_1:1-4.shelf2 -p 1

b. Repeat the process for the second node at the local site, systematically assigning the local disk
shelves to pool 0 and the remote disk shelves to pool 1:
disk assign -shelf local-switch-name:shelf-name.port -p pool

Example
If storage controller Controller_A_2 has four shelves, you issue the following commands:

*> disk assign -shelf FC_switch_A_1:1-4.shelf3 -p 0

*> disk assign -shelf FC_switch_B_1:1-4.shelf4 -p 1

*> disk assign -shelf FC_switch_A_1:1-4.shelf3 -p 0

*> disk assign -shelf FC_switch_B_1:1-4.shelf4 -p 1

3. Assign the disk shelves to the nodes located at the second site (site B):
126 | MetroCluster Installation and Configuration Guide

Disk shelves at the same site as the node are assigned to pool 0 and disk shelves located at the
partner site are assigned to pool 1.
You should assign an equal number of shelves to each pool.

a. On the first node at the remote site, systematically assign its local disk shelves to pool 0 and
its remote disk shelves to pool 1:
disk assign -shelf local-switch-nameshelf-name -p pool

Example
If storage controller Controller_B_1 has four shelves, you issue the following commands:

*> disk assign -shelf FC_switch_B_1:1-5.shelf1 -p 0

*> disk assign -shelf FC_switch_B_1:1-5.shelf2 -p 0

*> disk assign -shelf FC_switch_A_1:1-5.shelf1 -p 1

*> disk assign -shelf FC_switch_A_1:1-5.shelf2 -p 1

b. Repeat the process for the second node at the remote site, systematically assigning its local
disk shelves to pool 0 and its remote disk shelves to pool 1:
disk assign -shelf shelf-name -p pool

Example
If storage controller Controller_B_2 has four shelves, you issue the following commands:

*> disk assign -shelf FC_switch_B_1:1-5.shelf3 -p 0

*> disk assign -shelf FC_switch_B_1:1-5.shelf4 -p 0

*> disk assign -shelf FC_switch_A_1:1-5.shelf3 -p 1

*> disk assign -shelf FC_switch_A_1:1-5.shelf4 -p 1

4. Confirm the shelf assignments:

storage show shelf

5. Exit Maintenance mode:

halt

6. Display the boot menu:

boot_ontap menu

7. On each node, select option 4 to initialize all disks.

Assigning disk ownership in AFF systems

If you are using AFF systems and the nodes do not have the disks (SSDs) correctly assigned, you
must assign half the disks on each shelf to one local node and the other half of the disks to its HA
partner node. You will create a configuration in which each node has the same number of disks in its
local and remote disk pools.

Before you begin

The storage controllers must be in Maintenance mode.

About this task

This task is not required if disks were correctly assigned when received from the factory.
 Configuring the MetroCluster software in Data ONTAP | 127

Note: Pool 0 always contains the disks that are found at the same site as the storage system that
owns them.
Pool 1 always contains the disks that are remote to the storage system that owns them.

Steps

1. If you have not done so, boot each system into Maintenance mode.

2. Assign the disks to the nodes located at the first site (site A):
You should assign an equal number of disks to each pool.

a. On the first node, systematically assign half the disks on each shelf to pool 0 and the other
half to the HA partner's pool 0:
disk assign -disk disk-name -p pool -n number-of-disks

Example
If storage controller Controller_A_1 has four shelves, each with 8 SSDs, you issue the
following commands:

*> disk assign -shelf FC_switch_A_1:1-4.shelf1 -p 0 -n 4

*> disk assign -shelf FC_switch_A_1:1-4.shelf2 -p 0 -n 4

*> disk assign -shelf FC_switch_B_1:1-4.shelf1 -p 1 -n 4

*> disk assign -shelf FC_switch_B_1:1-4.shelf2 -p 1 -n 4

b. Repeat the process for the second node at the local site, systematically assigning half the disks
on each shelf to pool 1 and the other half to the HA partner's pool 1:
disk assign -disk disk-name -p pool

Example
If storage controller Controller_A_1 has four shelves, each with 8 SSDs, you issue the
following commands:

*> disk assign -shelf FC_switch_A_1:1-4.shelf3 -p 0 -n 4

*> disk assign -shelf FC_switch_B_1:1-4.shelf4 -p 1 -n 4

*> disk assign -shelf FC_switch_A_1:1-4.shelf3 -p 0 -n 4

*> disk assign -shelf FC_switch_B_1:1-4.shelf4 -p 1 -n 4

3. Assign the disks to the nodes located at the second site (site B):
You should assign an equal number of disks to each pool.

a. On the first node at the remote site, systematically assign half the disks on each shelf to pool 0
and the other half to the HA partner's pool 0:
disk assign -disk disk-name -p pool

Example
If storage controller Controller_B_1 has four shelves, each with 8 SSDs, you issue the
following commands:
128 | MetroCluster Installation and Configuration Guide

*> disk assign -shelf FC_switch_B_1:1-5.shelf1 -p 0 -n 4

*> disk assign -shelf FC_switch_B_1:1-5.shelf2 -p 0 -n 4

*> disk assign -shelf FC_switch_A_1:1-5.shelf1 -p 1 -n 4

*> disk assign -shelf FC_switch_A_1:1-5.shelf2 -p 1 -n 4

b. Repeat the process for the second node at the remote site, systematically assigning half the
disks on each shelf to pool 1 and the other half to the HA partner's pool 1:
disk assign -disk disk-name -p pool

Example
If storage controller Controller_B_2 has four shelves, each with 8 SSDs, you issue the
following commands:

*> disk assign -shelf FC_switch_B_1:1-5.shelf3 -p 0 -n 4

*> disk assign -shelf FC_switch_B_1:1-5.shelf4 -p 0 -n 4

*> disk assign -shelf FC_switch_A_1:1-5.shelf3 -p 1 -n 4

*> disk assign -shelf FC_switch_A_1:1-5.shelf4 -p 1 -n 4

4. Confirm the disk assignments:

storage show disk

5. Exit Maintenance mode:

halt

6. Display the boot menu:

boot_ontap menu

7. On each node, select option 4 to initialize all disks.

Verifying disk assignment in Maintenance mode in a two-

node configuration
Before fully booting the system to Data ONTAP, you can optionally boot to Maintenance mode and
verify the disk assignment on the nodes. The disks should be assigned to create a fully symmetric
active-active configuration, where each node and each pool have an equal number of disks assigned
to them.

About this task

New MetroCluster systems have disk assignment completed prior to shipment.
The following table shows example pool assignments for a MetroCluster configuration. Disks are
assigned to pools on a per-shelf basis.
 Configuring the MetroCluster software in Data ONTAP | 129

Disk shelf (example At site... Belongs to... And is assigned to

name)... that node's...

Disk shelf 1 Site A Node A 1 Pool 0

(shelf_A_1_1)
Disk shelf 2
(shelf_A_1_3)
Disk shelf 3 Node B 1 Pool 1
(shelf_B_1_1)
Disk shelf 4
(shelf_B_1_3)
Disk shelf 9 Site B Node B 1 Pool 0
(shelf_B_1_2)
Disk shelf 10
(shelf_B_1_4)
Disk shelf 11 Node A 1 Pool 1
(shelf_A_1_2)
Disk shelf 12
(shelf_A_1_4)

Steps

1. Confirm the shelf assignments:

disk show v

2. If necessary, you can explicitly assign disks on the attached disk shelves to the appropriate pool
with the disk assign command.
Using wildcards in the command enables you to assign all the disks on a disk shelf with one
command.

3. Show the disk shelf IDs and bays for each disk:
storage show disk x

Verifying the HA state of components is mcc or mcc-2n in

Maintenance mode
In a MetroCluster configuration, you must set the HA state of the controller and chassis components
to mcc or mcc-2n so they boot up properly. For systems received from the factory, this value is pre-
configured and you do not need to verify it.

Before you begin

The system must be in Maintenance mode.

About this task

This task is not required on systems received from the factory.
This task must be performed on each node.
130 | MetroCluster Installation and Configuration Guide

Steps

1. In Maintenance mode, display the HA state of the controller module and chassis:
ha-config show

The correct HA state depends on whether you have a four-node or two-node MetroCluster
configuration:

Number of controllers in the MetroCluster HA state for all components should be...
configuration
Four mcc
Two mcc-2n

2. If the displayed system state of the controller is not mcc, set the HA state for the controller
module to mcc or mcc-2n:

Number of controllers in Command

the MetroCluster
configuration
Four ha-config modify controller mcc
Two ha-config modify controller mcc-2n

3. If the displayed system state of the chassis is not mcc, set the HA state for the chassis to mcc or
mcc-2n:

Number of controllers in Command

the MetroCluster
configuration
Four ha-config modify chassis mcc
Two ha-config modify chassis mcc-2n

4. Boot the node to Data ONTAP:

boot_ontap

5. Repeat these steps on each node in the MetroCluster configuration.

Setting up Data ONTAP

You must set up Data ONTAP on each controller module. The procedure you use differs depending
on whether you have a two-node or four-node configuration.

Choices
Running System Setup in a four-node MetroCluster configuration on page 130
Setting up the clusters in a two-node MetroCluster configuration on page 133

Running System Setup in a four-node MetroCluster configuration

After you boot up each node, you are prompted to run the System Setup tool to perform basic node
and cluster configuration. After configuring the cluster, you return to the Data ONTAP CLI to create
aggregates and create the MetroCluster configuration.

Before you begin

You must have cabled the MetroCluster configuration.
 Configuring the MetroCluster software in Data ONTAP | 131

You must not have configured the Service Processor prior to performing this task.

About this task

This task is for four-node MetroCluster configurations using native NetApp storage.
New MetroCluster systems are pre-configured; you do not need to perform these steps. However, you
should configure the AutoSupport tool.
This task must be performed on both clusters in the MetroCluster configuration.
This procedure uses the System Setup tool. If desired, you can use the CLI cluster setup wizard.
Clustered Data ONTAP 8.3 Software Setup Guide

Steps

1. If you have not already done so, power up each node and let them boot up.
If the system is in Maintenance mode, issue the halt command to exit Maintenance mode, and
then issue the following command from the LOADER prompt:
boot_ontap

Example
The output should be similar to the following:

Welcome to node setup

You can enter the following commands at any time:

"help" or "?" - if you want to have a question clarified,
"back" - if you want to change previously answered questions, and
"exit" or "quit" - if you want to quit the setup wizard.
Any changes you made before quitting will be saved.

To accept a default or omit a question, do not enter a value.

.
.
.

2. Enable the AutoSupport tool by following the directions provided by the system.

3. Respond to the prompts to configure the node management interface.

Example
The prompts are similar to the following:

Enter the node management interface port: [e0M]:

Enter the node management interface IP address: 10.228.160.229
Enter the node management interface netmask: 225.225.252.0
Enter the node management interface default gateway: 10.228.160.1

4. If you have a four-node MetroCluster configuration, confirm that nodes are configured in high-
availability mode:
storage failover show -fields mode

If not, you must issue the following command on each node and reboot the node:
storage failover modify -mode ha -node localhost

This command configures high availability mode but does not enable storage failover. Storage
failover is automatically enabled when the MetroCluster configuration is performed later in the
configuration process.
132 | MetroCluster Installation and Configuration Guide

5. Confirm that you have four ports configured as cluster interconnects:

network port show

Example
The following example shows output for two controllers in cluster_A. If it is a two-node
MetroCluster configuration, the output will show only one node.

cluster_A::> network port show

Speed (Mbps)
Node Port IPspace Broadcast Domain Link MTU Admin/Oper
------ --------- ------------ ---------------- ----- ------- ------------
node_A_1
e0a Cluster Cluster up 1500 auto/1000
e0b Cluster Cluster up 1500 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
e0e Default Default up 1500 auto/1000
e0f Default Default up 1500 auto/1000
e0g Default Default up 1500 auto/1000
node_A_2
e0a Cluster Cluster up 1500 auto/1000
e0b Cluster Cluster up 1500 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
e0e Default Default up 1500 auto/1000
e0f Default Default up 1500 auto/1000
e0g Default Default up 1500 auto/1000
14 entries were displayed.

6. If you are creating a two-node switchless cluster (a cluster without cluster interconnect switches),
enable the switchless-cluster networking mode:

a. Change to the advanced privilege level:

set -privilege advanced

You can respond y when prompted to continue into advanced mode. The advanced mode
prompt appears (*>).

b. Enable switchless-cluster mode:

network options switchless-cluster modify -enabled true

c. Return to the admin privilege level:

set -privilege admin

7. Launch the System Setup tool as directed by the information that appears on the system console
after the initial boot.

8. Use the System Setup tool to configure each node and create the cluster, but do not create
aggregates.
Note: You create mirrored aggregates in later tasks.

After you finish

Return to the Data ONTAP command-line interface and complete the MetroCluster configuration by
performing the tasks that follow.
 Configuring the MetroCluster software in Data ONTAP | 133

Setting up the clusters in a two-node MetroCluster configuration

In a two-node MetroCluster configuration, you must boot up the node, exit the Node Setup wizard,
and use the cluster setup wizard to configure the node into a single-node cluster.

Before you begin

You must not have configured the Service Processor.

About this task

This task is for two-node MetroCluster configurations using native NetApp storage.
New MetroCluster systems are preconfigured; you do not need to perform these steps. However, you
should configure AutoSupport.
This task must be performed on both clusters in the MetroCluster configuration.
For more general information about setting up Data ONTAP, see Clustered Data ONTAP 8.3
Software Setup Guide.

Steps

1. Power on the first node.

The node boots, and then the Node Setup wizard starts on the console, informing you that
AutoSupport will be enabled automatically.

Welcome to node setup.

You can enter the following commands at any time:

"help" or "?" - if you want to have a question clarified,
"back" - if you want to change previously answered questions, and
"exit" or "quit" - if you want to quit the setup wizard.
Any changes you made before quitting will be saved.

To accept a default or omit a question, do not enter a value.

This system will send event messages and weekly reports to NetApp
Technical
Support. To disable this feature, enter "autosupport modify -support
disable"
within 24 hours. Enabling AutoSupport can significantly speed problem
determination and resolution should a problem occur on your system.
For
further information on AutoSupport, see:
http://support.netapp.com/autosupport/

Type yes to confirm and continue {yes}:

2. Because you are using the CLI to set up the cluster, exit the Node Setup wizard; Node Setup
wizard:
exit

The Node Setup would be used to configure the node's node management interface for use with
System Setup
The Node Setup wizard exits, and a login prompt appears, warning that you have not completed
the setup tasks:
134 | MetroCluster Installation and Configuration Guide

Exiting the node setup wizard. Any changes you made have been saved.

Warning: You have exited the node setup wizard before completing all
of the tasks. The node is not configured. You can complete node setup
by typing
"node setup" in the command line interface.

login:

3. Log in to the admin account by using the admin user name.

4. Start the Cluster Setup wizard:

cluster setup

::> cluster setup

Welcome to the cluster setup wizard.

You can enter the following commands at any time:

"help" or "?" - if you want to have a question clarified,
"back" - if you want to change previously answered questions, and
"exit" or "quit" - if you want to quit the cluster setup wizard.
Any changes you made before quitting will be saved.

You can return to cluster setup at any time by typing "cluster setup".
To accept a default or omit a question, do not enter a value.

Do you want to create a new cluster or join an existing cluster?

{create, join}:

5. Create a new cluster:

create

6. Accept the system defaults by pressing Enter, or enter your own values by typing no and then
pressing Enter.

7. Follow the prompts to complete the Cluster Setup wizard, pressing Enter to accept the default
values or typing your own values and then pressing Enter.
The default values are determined automatically based on your platform and network
configuration.

8. After you complete the Cluster Setup wizard and it exits, verify that the cluster is active and the
first node is healthy:
cluster show

Example
The following example shows a cluster in which the first node (cluster1-01) is healthy and
eligible to participate:

cluster1::> cluster show

Node Health Eligibility
--------------------- ------- ------------
cluster1-01 true true

If it becomes necessary to change any of the settings you entered for the admin SVM or node
SVM, you can access the Cluster Setup wizard by using the cluster setup command.
 Configuring the MetroCluster software in Data ONTAP | 135

Configuring the clusters into a MetroCluster configuration

You must mirror the root aggregates, create a mirrored data aggregate, and then issue the command to
implement the MetroCluster operations.

Peering the clusters

The clusters in the MetroCluster configuration must be in a peer relationship so that they can
communicate with each other and perform the data mirroring essential to MetroCluster disaster
recovery.

Before you begin

For systems received from the factory, the cluster peering is configured and you do not need to
manually peer the clusters.

Steps
1. Reviewing the preconfigured SVMs and LIFs on page 135
2. Manually peering the clusters on page 137

Related concepts
Considerations when using dedicated ports on page 13
Considerations when sharing data ports on page 13

Related references
Prerequisites for cluster peering on page 12

Related information
Clustered Data ONTAP 8.3 Data Protection Guide
Clustered Data ONTAP 8.3 Cluster Peering Express Guide

Reviewing the preconfigured SVMs and LIFs

The MetroCluster nodes are preconfigured with SVMs and LIFs. You need to be aware of these
settings as you perform your site-specific configuration.

LIFs for Cluster Storage Virtual Machine (SVM)

On the 32xx platforms, the symbol # indicates that the slot location might vary depending on system
configuration:

For 32xx systems with an IOXM, slot 2 is used.

For 32xx systems with two controllers in the chassis, or a controller and a blank, slot 1 is used.
136 | MetroCluster Installation and Configuration Guide

LIF Network Node Port

address/
32xx 62xx 8020 8040, 8060, or
mask
8080
clusterA-01_clus1 169.254.x.x/ cluster e#a e0c e0a e0a
16 A-01
clusterA-01_clus2 169.254.x.x/ e#b e0e e0b e0b
16
clusterA-01_clus3 169.254.x.x/ - n/a - n/a - - n/a - e0c
16 -
clusterA-01_clus4 169.254.x.x/ - n/a - n/a - - n/a - e0d
16 -
clusterA-02_clus1 169.254.x.x/ cluster e#a e0c e0a e0a
16 A-02
clusterA-02_clus2 169.254.x.x/ e#b e0e e0b e0b
16
clusterA-02_clus3 169.254.x.x/ - n/a - n/a - - n/a - e0c
16 -
clusterA-02_clus4 169.254.x.x/ - n/a - n/a - - n/a - e0d
16 -

LIFs for clusterA Storage Virtual Machine (SVM)

LIF Network Node Port

address/
32xx 62xx 8020 8040, 8060, or
mask
8080
clusterA-01-ic1 192.168.224. cluster e0a e0a e0e e0i
221/24 A-01
clusterA-01-ic2 192.168.224. e0b e0b e0f e0j
223/24
clusterA-01_mgmt1 10.10.10.11/ e0M e0M e0M e0M
24
clusterA-02-ic1 192.168.224. cluster e0a e0a e0e e0i
222/24 A-02
clusterA-02-ic2 192.168.224. e0b e0b e0f e0j
224/24
clusterA-02_mgmt1 10.10.10.12/ e0M e0M e0M e0M
24
cluster_mgmt 10.10.10.9/2 cluster e0a e0a e0e e0i
4 A-01
 Configuring the MetroCluster software in Data ONTAP | 137

LIFs for clusterB Storage Virtual Machine (SVM)

LIF Network Node Port

address/
32xx 62xx 8020 8040, 8060, or
mask
8080
clusterB-01-ic1 192.168.224. cluster e0a e0a e0e e0i
225/24 B-01
clusterB-01-ic2 192.168.224. e0b e0b e0f e0j
227/24
clusterB-01_mgmt1 10.10.10.13/ e0M e0M e0M e0M
24
clusterB-02-ic1 192.168.224. cluster e0a e0a e0e e0i
226/24 B-02
clusterB-02-ic2 192.168.224. e0b e0b e0f e0j
228/24
clusterB-02_mgmt1 10.10.10.14/ e0M e0M e0M e0M
24
cluster_mgmt 10.10.10.10/ cluster e0a e0a e0e e0i
24 B-01

Manually peering the clusters

To manually create the cluster peering relationship, you must decide whether to use dedicated LIFs,
configure the LIFs and then enable the relationship.

Before you begin

For systems received from the factory, the cluster peering is configured and you do not need to
manually peer the clusters.

Configuring intercluster LIFs

You must create intercluster LIFs on ports used for communication between the MetroCluster partner
clusters. You can use dedicated ports or ports that also have data traffic.

Choices
Configuring intercluster LIFs to use dedicated intercluster ports on page 137
Configuring intercluster LIFs to share data ports on page 141

Configuring intercluster LIFs to use dedicated intercluster ports

Configuring intercluster LIFs to use dedicated data ports allows greater bandwidth than using shared
data ports on your intercluster networks for cluster peer relationships.

About this task

Creating intercluster LIFs that use dedicated ports involves creating a failover group for the dedicated
ports and assigning LIFs to those ports. In this procedure, a two-node cluster exists in which each
node has two data ports that you have added, e0e and e0f. These ports are ones you will dedicate for
intercluster replication and currently are in the default IPspace. These ports will be grouped together
as targets for the intercluster LIFs you are configuring. You must configure intercluster LIFs on the
peer cluster before you can create cluster peer relationships. In your own environment, you would
138 | MetroCluster Installation and Configuration Guide

replace the ports, networks, IP addresses, subnet masks, and subnets with those specific to your
environment.

Steps

1. List the ports in the cluster by using network port show command.

Example

cluster01::> network port show

Speed (Mbps)
Node Port IPspace Broadcast Domain Link MTU Admin/Oper
------ --------- ------------ ---------------- ----- ------- ------------
cluster01-01
e0a Cluster Cluster up 1500 auto/1000
e0b Cluster Cluster up 1500 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
e0e Default Default up 1500 auto/1000
e0f Default Default up 1500 auto/1000
cluster01-02
e0a Cluster Cluster up 1500 auto/1000
e0b Cluster Cluster up 1500 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
e0e Default Default up 1500 auto/1000
e0f Default Default up 1500 auto/1000

2. Determine whether any of the LIFs are using ports that are dedicated for replication by using the
network interface show command.

Example
Ports e0e and e0f do not appear in the following output; therefore, they do not have any LIFs
located on them:

cluster01::> network interface show -fields home-port,curr-port

vserver lif home-port curr-port
------- -------------------- --------- ---------
Cluster cluster01-01_clus1 e0a e0a
Cluster cluster01-01_clus2 e0b e0b
Cluster cluster01-02_clus1 e0a e0a
Cluster cluster01-02_clus2 e0b e0b
cluster01
cluster_mgmt e0c e0c
cluster01
cluster01-01_mgmt1 e0c e0c
cluster01
cluster01-02_mgmt1 e0c e0c

3. If a LIF is using a port that you want dedicated to intercluster connectivity, migrate the LIF to a
different port.

a. Migrate the LIF to another port by using the network interface migrate command.

Example
The following example assumes that the data LIF named cluster01_data01 uses port e0e and
you want only an intercluster LIF to use that port:

cluster01::> network interface migrate -vserver cluster01

-lif cluster01_data01 -dest-node cluster01-01 -dest-port e0d

b. You might need to modify the migrated LIF home port to reflect the new port where the LIF
should reside by using the network interface modify command:
 Configuring the MetroCluster software in Data ONTAP | 139

Example

cluster01::> network interface modify -vserver cluster01

-lif cluster01_data01 -home-node cluster01-01 -home-port e0d

4. Group the ports that you will use for the intercluster LIFs by using the network interface
failover-groups create command.

Example

cluster01::> network interface failover-groups create -vserver cluster01

-failover-group intercluster01 -targets cluster01-01:e0e,cluster01-01:e0f,
cluster01-02:e0e,cluster01-02:e0f

5. Display the failover-group that you created by using the network interface failover-
groups show command.

Example

cluster01::> network interface failover-groups show

Failover
Vserver Group Targets
---------------- ---------------- --------------------------------------------
Cluster
Cluster
cluster01-01:e0a, cluster01-01:e0b,
cluster01-02:e0a, cluster01-02:e0b
cluster01
Default
cluster01-01:e0c, cluster01-01:e0d,
cluster01-02:e0c, cluster01-02:e0d,
cluster01-01:e0e, cluster01-01:e0f
cluster01-02:e0e, cluster01-02:e0f
intercluster01
cluster01-01:e0e, cluster01-01:e0f
cluster01-02:e0e, cluster01-02:e0f

6. Create an intercluster LIF on the admin SVM cluster01 by using the network interface
create command.

Example
This example uses the LIF naming convention adminSVMname_icl# for the intercluster LIF:

cluster01::> network interface create -vserver cluster01 -lif cluster01_icl01 -role

intercluster -home-node cluster01-01 -home-port e0e
-address 192.168.1.201 -netmask 255.255.255.0 -failover-group intercluster01

cluster01::> network interface create -vserver cluster01 -lif cluster01_icl02 -role

intercluster -home-node cluster01-02 -home-port e0e
-address 192.168.1.202 -netmask 255.255.255.0 -failover-group intercluster01

7. Verify that the intercluster LIFs were created properly by using the network interface show
command.

Example

cluster01::> network interface show

Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
Cluster
cluster01-01_clus_1
up/up 192.168.0.xxx/24 cluster01-01 e0a true
cluster01-01_clus_2
up/up 192.168.0.xxx/24 cluster01-01 e0b true
cluster01-02_clus_1
up/up 192.168.0.xxx/24 cluster01-01 e0a true
cluster01-02_clus_2
up/up 192.168.0.xxx/24 cluster01-01 e0b true
cluster01
cluster_mgmt up/up 192.168.0.xxx/24 cluster01-01 e0c true
140 | MetroCluster Installation and Configuration Guide

cluster01_icl01
up/up 192.168.1.201/24 cluster01-01 e0e true
cluster01_icl02
up/up 192.168.1.202/24 cluster01-02 e0e true
cluster01-01_mgmt1
up/up 192.168.0.xxx/24 cluster01-01 e0c true
cluster01-02_mgmt1
up/up 192.168.0.xxx/24 cluster01-02 e0c true

8. Verify that the intercluster LIFs are configured for redundancy by using the network
interface show command with the -role intercluster and -failover parameters.

Example
The LIFs in this example are assigned the e0e home port on each node. If the e0e port fails, the
LIF can fail over to the e0f port.

cluster01::> network interface show -role intercluster failover

Logical Home Failover Failover
Vserver Interface Node:Port Policy Group
-------- --------------- --------------------- --------------- --------
cluster01-01
cluster01-01_icl01 cluster01-01:e0e local-only intercluster01
Failover Targets: cluster01-01:e0e,
cluster01-01:e0f
cluster01-01_icl02 cluster01-02:e0e local-only intercluster01
Failover Targets: cluster01-02:e0e,
cluster01-02:e0f

9. Display the routes in the cluster by using the network route show command to determine
whether intercluster routes are available or you must create them.
Creating a route is required only if the intercluster addresses in both clusters are not on the same
subnet and a specific route is needed for communication between the clusters.

Example
In this example, no intercluster routes are available:

cluster01::> network route show

Vserver Destination Gateway Metric
--------- --------------- --------------- ------
Cluster
0.0.0.0/0 192.168.0.1 20
cluster01
0.0.0.0/0 192.168.0.1 10

10. If communication between intercluster LIFs in different clusters requires routing, create an
intercluster route by using the network route create command.
The gateway of the new route should be on the same subnet as the intercluster LIF.

Example
In this example, 192.168.1.1 is the gateway address for the 192.168.1.0/24 network. If the
destination is specified as 0.0.0.0/0, then it becomes a default route for the intercluster network.

cluster01::> network route create -vserver cluster01

-destination 0.0.0.0/0 -gateway 192.168.1.1 -metric 40

11. Verify that you created the routes correctly by using the network route show command.

Example

cluster01::> network route show

Vserver Destination Gateway Metric
--------- --------------- --------------- ------
Cluster
 Configuring the MetroCluster software in Data ONTAP | 141

0.0.0.0/0 192.168.0.1 20
cluster01
0.0.0.0/0 192.168.0.1 10
0.0.0.0/0 192.168.1.1 40

12. Repeat these steps to configure intercluster networking in the peer cluster.

13. Verify that the ports have access to the proper subnets, VLANs, and so on.
Dedicating ports for replication in one cluster does not require dedicating ports in all clusters; one
cluster might use dedicated ports, while the other cluster shares data ports for intercluster
replication.

Related concepts
Considerations when using dedicated ports on page 13

Configuring intercluster LIFs to share data ports

Configuring intercluster LIFs to share data ports enables you to use existing data ports to create
intercluster networks for cluster peer relationships. Sharing data ports reduces the number of ports
you might need for intercluster networking.

About this task

Creating intercluster LIFs that share data ports involves assigning LIFs to existing data ports. In this
procedure, a two-node cluster exists in which each node has two data ports, e0c and e0d, and these
data ports are in the default IPspace. These are the two data ports that are shared for intercluster
replication. You must configure intercluster LIFs on the peer cluster before you can create cluster
peer relationships. In your own environment, you replace the ports, networks, IP addresses, subnet
masks, and subnets with those specific to your environment.

Steps

1. List the ports in the cluster by using the network port show command:

Example

cluster01::> network port show

Speed (Mbps)
Node Port IPspace Broadcast Domain Link MTU Admin/Oper
------ --------- ------------ ---------------- ----- ------- ------------
cluster01-01
e0a Cluster Cluster up 1500 auto/1000
e0b Cluster Cluster up 1500 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
cluster01-02
e0a Cluster Cluster up 1500 auto/1000
e0b Cluster Cluster up 1500 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000

2. Create an intercluster LIF on the admin SVM cluster01 by using the network interface
create command.

Example
This example uses the LIF naming convention
adminSVMname_icl#
for the intercluster LIF:

cluster01::> network interface create -vserver cluster01 -lif cluster01_icl01 -role

intercluster
-home-node cluster01-01 -home-port e0c -address 192.168.1.201 -netmask 255.255.255.0
142 | MetroCluster Installation and Configuration Guide

cluster01::> network interface create -vserver cluster01 -lif cluster01_icl02 -role

intercluster
-home-node cluster01-02 -home-port e0c -address 192.168.1.202 -netmask 255.255.255.0

3. Verify that the intercluster LIFs were created properly by using the network interface show
command with the -role intercluster parameter:

Example

cluster01::> network interface show role intercluster

Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
cluster01
cluster01_icl01
up/up 192.168.1.201/24 cluster01-01 e0c true
cluster01_icl02
up/up 192.168.1.202/24 cluster01-02 e0c true

4. Verify that the intercluster LIFs are configured to be redundant by using the network
interface show command with the -role intercluster and -failover parameters.

Example
The LIFs in this example are assigned the e0c port on each node. If the e0c port fails, the LIF can
fail over to the e0d port.

cluster01::> network interface show -role intercluster failover

Logical Home Failover Failover
Vserver Interface Node:Port Policy Group
-------- --------------- --------------------- --------------- --------
cluster01
cluster01_icl01 cluster01-01:e0c local-only 192.168.1.201/24
Failover Targets: cluster01-01:e0c,
cluster01-01:e0d
cluster01_icl02 cluster01-02:e0c local-only 192.168.1.201/24
Failover Targets: cluster01-02:e0c,
cluster01-02:e0d

5. Display the routes in the cluster by using the network route show command to determine
whether intercluster routes are available or you must create them.
Creating a route is required only if the intercluster addresses in both clusters are not on the same
subnet and a specific route is needed for communication between the clusters.

Example
In this example, no intercluster routes are available:

cluster01::> network route show

Vserver Destination Gateway Metric
--------- --------------- --------------- ------
Cluster
0.0.0.0/0 192.168.0.1 20
cluster01
0.0.0.0/0 192.168.0.1 10

6. If communication between intercluster LIFs in different clusters requires routing, create an

intercluster route by using the network route create command.
The gateway of the new route should be on the same subnet as the intercluster LIF.

Example
In this example, 192.168.1.1 is the gateway address for the 192.168.1.0/24 network. If the
destination is specified as 0.0.0.0/0, then it becomes a default route for the intercluster network.
 Configuring the MetroCluster software in Data ONTAP | 143

cluster01::> network route create -vserver cluster01

-destination 0.0.0.0/0 -gateway 192.168.1.1 -metric 40

7. Verify that you created the routes correctly by using the network route show command.

Example

cluster01::> network route show

Vserver Destination Gateway Metric
--------- --------------- --------------- ------
Cluster
0.0.0.0/0 192.168.0.1 20
cluster01
0.0.0.0/0 192.168.0.1 10
0.0.0.0/0 192.168.1.1 40

8. Repeat these steps on the cluster to which you want to connect.

Related concepts
Considerations when sharing data ports on page 13

Creating the cluster peer relationship

You create the cluster peer relationship using a set of intercluster logical interfaces to make
information about one cluster available to the other cluster for use in cluster peering applications.

Before you begin

Intercluster LIFs should be created in the IPspaces of both clusters you want to peer.

You should ensure that the intercluster LIFs of the clusters can route to each other.

If there are different administrators for each cluster, the passphrase used to authenticate the
cluster peer relationship should be agreed upon.

About this task

If you created intercluster LIFs in a nondefault IPspace, you need to designate the IPspace when you
create the cluster peer.

Steps

1. Create the cluster peer relationship on each cluster by using the cluster peer create
command.
The passphrase that you use is not displayed as you type it.
If you created a nondefault IPspace to designate intercluster connectivity, you use the ipspace
parameter to select that IPspace.

Example
In the following example, cluster01 is peered with a remote cluster named cluster02. Cluster01 is
a two-node cluster that has one intercluster LIF per node. The IP addresses of the intercluster
LIFs created in cluster01 are 192.168.2.201 and 192.168.2.202. Similarly, cluster02 is a two-node
cluster that has one intercluster LIF per node. The IP addresses of the intercluster LIFs created in
cluster02 are 192.168.2.203 and 192.168.2.204. These IP addresses are used to create the cluster
peer relationship.
144 | MetroCluster Installation and Configuration Guide

cluster01::> cluster peer create -peer-addrs

192.168.2.203,192.168.2.204
Please type the passphrase:
Please type the passphrase again:

cluster02::> cluster peer create -peer-addrs

192.168.2.201,192.168.2.202
Please type the passphrase:
Please type the passphrase again:

If DNS is configured to resolve host names for the intercluster IP addresses, you can use host
names in the peer-addrs option. It is not likely that intercluster IP addresses frequently
change; however, using host names allows intercluster IP addresses to change without having to
modify the cluster peer relationship.

Example
In the following example, an IPspace called IP01A was created on cluster01 for intercluster
connectivity. The IP addresses used in the previous example are used in this example to create the
cluster peer relationship.

cluster01::> cluster peer create -peer-addrs

192.168.2.203,192.168.2.204
-ipspace IP01A
Please type the passphrase:
Please type the passphrase again:

cluster02::> cluster peer create -peer-addrs

192.168.2.201,192.168.2.202
Please type the passphrase:
Please type the passphrase again:

2. Display the cluster peer relationship by using the cluster peer show command with the -
instance parameter.
Displaying the cluster peer relationship verifies that the relationship was established successfully.

Example

cluster01::> cluster peer show instance

Peer Cluster Name: cluster02
Remote Intercluster Addresses: 192.168.2.203,192.168.2.204
Availability: Available
Remote Cluster Name: cluster02
Active IP Addresses: 192.168.2.203,192.168.2.204
Cluster Serial Number: 1-80-000013

3. Preview the health of the nodes in the peer cluster by using the cluster peer health show
command.
Previewing the health checks the connectivity and status of the nodes on the peer cluster.

Example

cluster01::> cluster peer health show

Node cluster-Name Node-Name
Ping-Status RDB-Health Cluster-Health Avail
---------- --------------------------- --------- --------------- --------
cluster01-01
cluster02 cluster02-01
Data: interface_reachable
ICMP: interface_reachable true true true
cluster02-02
Data: interface_reachable
ICMP: interface_reachable true true true
cluster01-02
 Configuring the MetroCluster software in Data ONTAP | 145

cluster02 cluster02-01
Data: interface_reachable
ICMP: interface_reachable true true true
cluster02-02
Data: interface_reachable
ICMP: interface_reachable true true true

Mirroring the root aggregates

You must mirror the root aggregates to ensure data protection.

Steps

1. Mirror the root aggregate:

storage aggregate mirror aggr_ID

Example
The following command mirrors the root aggregate for controller_A_1:

controller_A_1::> storage aggregate mirror aggr0_controller_A_1

This creates an aggregate with a local plex located at the local MetroCluster site and a remote
plex located at the remote MetroCluster site.

2. Repeat the previous steps for each node in the MetroCluster configuration.

Creating a mirrored data aggregate on each node

You must create a mirrored data aggregate on each node in the DR group.

Before you begin

You should know what drives or array LUNs will be used in the new aggregate.

If you have multiple drive types in your system (heterogeneous storage), you should understand
how you can ensure that the correct drive type is selected.

About this task

Drives and array LUNs are owned by a specific node; when you create an aggregate, all drives in
that aggregate must be owned by the same node, which becomes the home node for that
aggregate.

Aggregate names should conform to the naming scheme you determined when you planned your
MetroCluster configuration.

The Clustered Data ONTAP Data Protection Guide contains more information about mirroring
aggregates.

Steps

1. Display a list of available spares:

storage disk show -spare -owner node_name

2. Create the aggregate by using the storage aggregate create -mirror true command.
If you are logged in to the cluster on the cluster management interface, you can create an
aggregate on any node in the cluster. To ensure that the aggregate is created on a specific node,
use the -node parameter or specify drives that are owned by that node.
You can specify the following options:
146 | MetroCluster Installation and Configuration Guide

Aggregate's home node (that is, the node that owns the aggregate in normal operation)

List of specific drives or array LUNs that are to be added to the aggregate

Number of drives to include

Checksum style to use for the aggregate

Type of drives to use

Size of drives to use

Drive speed to use

RAID type for RAID groups on the aggregate

Maximum number of drives or array LUNs that can be included in a RAID group

Whether drives with different RPM are allowed

For more information about these options, see the storage aggregate create man page.

Example
The following command creates a mirrored aggregate with 10 disks:

controller_A_1::> storage aggregate create aggr1_controller_A_1 -

diskcount 10 -node controller_A_1 -mirror true
[Job 15] Job is queued: Create aggr1_controller_A_1.
[Job 15] The job is starting.
[Job 15] Job succeeded: DONE

3. Verify the RAID group and drives of your new aggregate:

storage aggregate show-status -aggregate aggregate-name

Implementing the MetroCluster configuration

You must run the metrocluster configure command to start data protection in the MetroCluster
configuration.

Before you begin

There must be at least two nonroot mirrored data aggregates on each cluster, and all aggregates
must be mirrored.
You can verify this with the storage aggregate show command.

The ha-config state of the controllers and chassis must be mcc.

This state is preconfigured on systems shipped from the factory.

About this task

You issue the metrocluster configure command once, on any of the nodes, to enable the
MetroCluster configuration. You do not need issue the command on each of the sites or nodes, and it
does not matter which node or site you choose to issue the command on.
The metrocluster configure command automatically pairs the two nodes with lowest system
IDs in each of the two clusters as DR partners. In a four-node MetroCluster, there are two DR partner
pairs. The second DR pair is created from the two nodes with higher system IDs.
 Configuring the MetroCluster software in Data ONTAP | 147

Steps

1. Enter the metrocluster configure command in the following format, depending on whether
you .

If your MetroCluster Then take these steps:

configuration will have..
Multiple data aggregates From any node's prompt, perform the MetroCluster configuration
operation:
This is the best practice.
metrocluster configure node-name
A single data aggregate
a. From any node's prompt, change to the advanced privilege level:
set -privilege advanced
You need to respond with y when prompted to continue into
advanced mode and see the advanced mode prompt (*>).

b. Perform the MetroCluster configuration operation with the -allow-

with-one-aggregate true parameter:
metrocluster configure -allow-with-one-
aggregate true node-name

c. Return to the admin privilege level:

set -privilege admin

Example
The following command enables MetroCluster configuration on all nodes in the DR group that
contains controller_A_1:

controller_A_1::*> metrocluster configure -node-name controller_A_1

[Job 121] Job succeeded: Configure is successful.

2. Check the networking status on site A:

network port show

Example

cluster_A::> network port show

Speed (Mbps)
Node Port IPspace Broadcast Domain Link MTU Admin/Oper
------ --------- --------- ---------------- ----- ------- ------------
controller_A_1
e0a Cluster Cluster up 9000 auto/1000
e0b Cluster Cluster up 9000 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
e0e Default Default up 1500 auto/1000
e0f Default Default up 1500 auto/1000
e0g Default Default up 1500 auto/1000
controller_A_2
e0a Cluster Cluster up 9000 auto/1000
e0b Cluster Cluster up 9000 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
e0e Default Default up 1500 auto/1000
e0f Default Default up 1500 auto/1000
e0g Default Default up 1500 auto/1000
14 entries were displayed.
148 | MetroCluster Installation and Configuration Guide

3. Confirm the MetroCluster configuration from both sites in the MetroCluster configuration:

a. Confirm the configuration from site A:

metrocluster show

Example

cluster_A::> metrocluster show

Cluster Configuration State Mode
------------------------- ------------------- -----------
Local: cluster_A configured normal
Remote: cluster_B configured normal

b. Confirm the configuration from site B:

metrocluster show

Example

cluster_B::> metrocluster show

Cluster Configuration State Mode
------------------------- ------------------- -----------
Local: cluster_B configured normal
Remote: cluster_A configured normal

Configuring the MetroCluster FC switches for health monitoring

In a fabric-attached MetroCluster configuration, you must perform some special configuration steps
to monitor the FC switches.

Steps

1. Issue the following command on each MetroCluster node:

storage switch add -switch-ipaddress ipaddress

This command must be repeated on all four switches in the MetroCluster configuration.

Example
The following example shows the command to add a switch with IP address of 10.10.10.10:

controller_A_1::> storage switch add -switch-ipaddress 10.10.10.10

2. Verify that all switches are properly configured:

storage switch show

It might take up to 15 minutes to reflect all data due to the 15-minute polling interval.

Example
The following example shows the command given to verify the MetroCluster's FC switches are
configured:

controller_A_1::> storage switch show

Fabric Switch Name Vendor Model Switch WWN Status
---------------- --------------- ------- ------------ ---------------- ------
1000000533a9e7a6 brcd6505-fcs40 Brocade Brocade6505 1000000533a9e7a6 OK
1000000533a9e7a6 brcd6505-fcs42 Brocade Brocade6505 1000000533d3660a OK
1000000533ed94d1 brcd6510-fcs44 Brocade Brocade6510 1000000533eda031 OK
 Configuring the MetroCluster software in Data ONTAP | 149

1000000533ed94d1 brcd6510-fcs45 Brocade Brocade6510 1000000533ed94d1 OK

4 entries were displayed.

controller_A_1::>

If the switch's worldwide name (WWN) is shown, the Data ONTAP health monitor is able to
contact and monitor the FC switch.

Related information
Clustered Data ONTAP 8.3 System Administration Guide

Checking the MetroCluster configuration

You can check that the components and relationships in the MetroCluster configuration are working
correctly. You should do a check after initial configuration, and after making any changes to the
MetroCluster configuration, and before a negotiated (planned) switchover or a switchback operation.
After you run the metrocluster check run command, you then display the results of the check
with various metrocluster check show commands.

About this task

If the metrocluster check run command is issued twice within a short time, on either or both
clusters, a conflict can occur and the command might not collect all data. Subsequent
metrocluster check show commands will not show the expected output.

Steps

1. Check the configuration:

metrocluster check run

Example
The command will run as a background job:

controller_A_1::> metrocluster check run

[Job 60] Job succeeded: Check is successful. Run "metrocluster check
show" command to view the results of this operation.

controller_A_1::> metrocluster check show

Last Checked On: 5/22/2015 12:23:28

Component Result
------------------- ---------
nodes ok
lifs ok
config-replication ok
aggregates ok
clusters ok
5 entries were displayed.

2. Display more detailed results from the most recent metrocluster check run command:
metrocluster check aggregate show
metrocluster check cluster show
metrocluster check config-replication show
metrocluster check lif show
metrocluster check node show
150 | MetroCluster Installation and Configuration Guide

The metrocluster check show commands show the results of the most recent
metrocluster check run command. You should always run the metrocluster check
run command prior to using the metrocluster check show commands to ensure that the
information displayed is current.

Example
The following example shows the metrocluster check aggregate show output for a
healthy four-node MetroCluster configuration:

controller_A_1::> metrocluster check aggregate show

Last Checked On: 8/5/2014 00:42:58

Node Aggregate Check Result

--------------------- --------------------- --------------------- ---------
controller_A_1 aggr0_controller_A_1_0
mirroring-status ok
disk-pool-allocation ok
controller_A_1_aggr1
mirroring-status ok
disk-pool-allocation ok
controller_A_2 aggr0_controller_A_2
mirroring-status ok
disk-pool-allocation ok
controller_A_2_aggr1
mirroring-status ok
disk-pool-allocation ok
controller_B_1 aggr0_controller_B_1_0
mirroring-status ok
disk-pool-allocation ok
controller_B_1_aggr1
mirroring-status ok
disk-pool-allocation ok
controller_B_2 aggr0_controller_B_2
mirroring-status ok
disk-pool-allocation ok
controller_B_2_aggr1
mirroring-status ok
disk-pool-allocation ok
16 entries were displayed.

The following example shows the metrocluster check cluster show output for a healthy
four-node MetroCluster configuration. It indicates that the clusters are ready to perform a
negotiated switchover if necessary.

Last Checked On: 5/22/2015 12:23:28

Cluster Check Result

--------------------- ------------------------------------ ---------
cluster_A
negotiated-switchover-ready ok
switchback-ready not-
applicable
job-schedules ok
licenses ok
cluster_B
negotiated-switchover-ready ok
switchback-ready not-
applicable
job-schedules ok
licenses ok
8 entries were displayed.

Related information
Clustered Data ONTAP 8.3 Physical Storage Management Guide
Clustered Data ONTAP 8.3 Network Management Guide
Clustered Data ONTAP 8.3 Data Protection Guide
 Configuring the MetroCluster software in Data ONTAP | 151

Checking for MetroCluster configuration errors with Config

Advisor
You can go to the NetApp Support Site and download the Config Advisor tool to check for common
configuration errors.

About this task

Config Advisor is a configuration validation and health check tool. You can deploy it at both secure
sites and non-secure sites for data collection and system analysis.
Note: Support for Config Advisor is limited, and available only online.

Steps

1. Go to NetApp Downloads: Config Advisor.

2. After running Config Advisor, review the tool's output and follow the recommendations in the
output to address any issues discovered.

Verifying local HA operation

If you have a four-node MetroCluster configuration, you should verify the operation of the local HA
pairs in the MetroCluster configuration. This is not required for two-node configurations.

About this task

Two-node MetroCluster configurations do not consist of local HA pairs and this task does not apply.
The examples in this task use standard naming conventions:

cluster_A

controller_A_1

controller_A_2
cluster_B

controller_B_1

controller_B_2

Steps

1. On cluster_A, perform a failover and giveback in both directions.

a. Confirm that storage failover is enabled:

storage failover show

Example
The output should indicate that takeover is possible for both nodes:

cluster_A::> storage failover show

Takeover
Node Partner Possible State Description
-------------- -------------- -------- ---------------------------
152 | MetroCluster Installation and Configuration Guide

controller_A_1 controller_A_2 true Connected to controller_A_2

controller_A_2 controller_A_1 true Connected to controller_A_1

2 entries were displayed.

b. Take over controller_A_2 from controller_A_1:

storage failover takeover controller_A_2

You can use the storage failover show-takeover command to monitor the progress of
the takeover operation.
c. Confirm that the takeover is complete:
storage failover show

Example
The output should indicate that controller_A_1 is in takeover state, meaning that it has taken
over its HA partner:

cluster_A::> storage failover show

Takeover
Node Partner Possible State Description
-------------- -------------- -------- -----------------
controller_A_1 controller_A_2 false In takeover

controller_A_2 controller_A_1 - Unknown

2 entries were displayed.

d. Give back controller_A_2:

storage failover giveback controller_A_2

You can use the storage failover show-giveback command to monitor the progress of
the giveback operation.

e. Confirm that storage failover has returned to a normal state:

storage failover show

Example
The output should indicate that takeover is possible for both nodes:

cluster_A::> storage failover show

Takeover
Node Partner Possible State Description
-------------- -------------- -------- ---------------------------
controller_A_1 controller_A_2 true Connected to controller_A_2

controller_A_2 controller_A_1 true Connected to controller_A_1

2 entries were displayed.

f. Repeat the previous substeps, this time taking over controller_A_1 from controller_A_2.

2. Repeat the preceding steps on cluster_B.

Related information
Clustered Data ONTAP 8.3 High-Availability Configuration Guide
 Configuring the MetroCluster software in Data ONTAP | 153

Verifying switchover, healing, and switchback

You should verify the switchover, healing, and switchback operations of the MetroCluster
configuration.

Step

1. Use the procedures for negotiated switchover, healing, and switchback in the Clustered Data
ONTAP 8.3 MetroCluster Management and Disaster Recovery Guide.

Installing the MetroCluster Tiebreaker software

You can download and install Tiebreaker software to monitor the two clusters and the connectivity
status between them from a third site. Doing so enables each partner in a cluster to distinguish
between an ISL failure (when inter-site links are down) from a site failure.

Before you begin

You must have a Linux host available that has network connectivity to both clusters in the
MetroCluster configuration.

Steps

1. Go to MetroCluster Tiebreaker Software Download page.

2. Follow the directions to download the Tiebreaker software and documentation.

Protecting configuration backup files

You can provide additional protection for the cluster configuration backup files by specifying a
remote URL (either HTTP or FTP) where the configuration backup files will be uploaded in addition
to the default locations in the local cluster.

Step

1. Set the URL of the remote destination for the configuration backup files:
system configuration backup settings modify URL-of-destination

Related information
Clustered Data ONTAP 8.3 System Administration Guide
154

Planning and installing a MetroCluster

configuration with array LUNs
If you are using array LUNs in your MetroCluster configuration, you must plan the installation and
follow the specific procedures for such a configuration. You can set up a MetroCluster configuration
with either a mix of array LUNs and native disk shelves or only array LUNs.
The MetroCluster configuration can have either two or four Data ONTAP nodes.
Although you must configure the four-node configuration only in fabric-attached mode, you can use
either stretch or fabric-attached modes for the two-node configuration, depending on your
requirements.

Planning for a MetroCluster configuration with array LUNs

Creating a detailed plan for your MetroCluster configuration helps you understand the unique
requirements for a MetroCluster configuration that uses LUNs on storage arrays. Installing a
MetroCluster configuration involves connecting and configuring a number of devices, which might
be done by different people. Therefore, the plan also helps you communicate with other people
involved in the installation.

Supported MetroCluster configuration with array LUNs

You can set up either a two-node or a four-node MetroCluster configuration with array LUNs. While
the four-node MetroCluster configuration must be fabric-attached, the two-node configuration can be
either stretch or fabric-attached.
The features supported on the MetroCluster configurations vary with the configuration types. The
following table lists the features supported on the different types of MetroCluster configurations with
array LUNs:

Feature Fabric-attached configurations Stretch configuration

Four-node Two-node Two-node
Number of controllers Four Two Two
Uses an FC switch Yes Yes Yes
storage fabric
Supports local HA Yes No No
Supports automatic No Yes Yes
switchover

Related concepts
Differences between the clustered Data ONTAP MetroCluster configurations on page 11
 Planning and installing a MetroCluster configuration with array LUNs | 155

Requirements for a MetroCluster configuration with array

LUNs
The Data ONTAP systems, storage arrays, and FC switches used in MetroCluster configurations must
meet the requirements for such types of configurations. In addition, you must also consider the
SyncMirror requirements for MetroCluster configurations with array LUNs.

Requirements for Data ONTAP systems using array LUNs in a MetroCluster

configuration
The Data ONTAP systems must be identified as supported for MetroCluster configurations.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.
Note: You must refer to the alert details associated with any configuration that you select in the
Interoperability Matrix.

All the Data ONTAP systems in a MetroCluster configuration must be of the same model.

FC-VI adapters must be installed into the appropriate slots for each Data ONTAP system,
depending on the model.
NetApp Hardware Universe

Requirements for storage arrays

The storage arrays must be identified as supported for MetroCluster configurations.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

The storage arrays in the MetroCluster configuration must be symmetric, which means the
following:

The two storage arrays must be from the same supported vendor family and have the same
firmware version installed.
FlexArray Virtualization Implementation Guide for NetApp E-Series Storage
FlexArray Virtualization Implementation Guide for Third-Party Storage
Disk types (for example, SATA, SSD, or SAS) used for mirrored storage must be the same on
both storage arrays.

The parameters for configuring storage arrays, such as RAID type and tiering, must be the
same across both sites.

Requirements for FC switches

The switches and switch firmware must be identified as supported for MetroCluster
configurations.
NetApp Interoperability Matrix Tool
156 | MetroCluster Installation and Configuration Guide

After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

Each fabric must have two FC switches.

Each Data ONTAP system must be connected to storage using redundant components so that
there is redundancy in case of device and path failures.
Your MetroCluster solution can use up to four ISLs, depending on the configuration.

For more information about basic switch configuration, ISL settings, and FC-VI configurations, see
Configuring the Cisco or Brocade FC switches manually on page 50.

SyncMirror requirements
SyncMirror is required for a MetroCluster configuration.

Two separate storage arrays, one at each site, are required for the mirrored storage.

Two sets of array LUNs are required.

One set is required for the aggregate on the local storage array (pool0) and another set is required
at the remote storage array for the mirror of the aggregate (the other plex of the aggregate, pool1).
The array LUNs must be of the same size for mirroring the aggregate.
Clustered Data ONTAP 8.3 Data Protection Guide

Installing and cabling the MetroCluster components in a

configuration with array LUNs
For setting up a MetroCluster configuration with array LUNs, you must cable the storage controllers
to the FC switches and cable the ISLs to link the sites. In addition, you must cable the storage arrays
to the FC switches.

Steps
1. Racking the hardware components in a MetroCluster configuration with array LUNs on page 157
2. Preparing a storage array for use with Data ONTAP systems on page 157
3. Switch ports required for a MetroCluster configuration with array LUNs on page 158
4. Cabling the FC-VI and HBA ports in a MetroCluster configuration with array LUNs on page 159
5. Cabling the ISLs between MetroCluster sites on page 164
6. Cabling the cluster interconnect in four-node configurations on page 165
7. Cabling the cluster peering connections on page 166
8. Cabling the HA interconnect, if necessary on page 166
9. Cabling the management and data connections on page 167
10. Cabling storage arrays to FC switches in a MetroCluster configuration on page 167
 Planning and installing a MetroCluster configuration with array LUNs | 157

Racking the hardware components in a MetroCluster configuration with

array LUNs
You must ensure that the hardware components required to set up a MetroCluster configuration with
array LUNs are properly racked.

About this task

You must perform this task at both the MetroCluster sites.

Steps

1. Plan the positioning of the MetroCluster components.

The rack space depends on the platform model of the storage controllers, the switch types, and the
number of disk shelf stacks in your configuration.

2. Properly ground yourself.

3. Install the storage controllers in the rack or cabinet.

Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems
4. Install the FC switches in the rack or cabinet.

Preparing a storage array for use with Data ONTAP systems

Before you can begin setting up Data ONTAP systems in a MetroCluster configuration with array
LUNs, the storage array administrator must prepare the storage for use with Data ONTAP.

Before you begin

The storage arrays, firmware, and switches that you plan to use in the configuration must be
supported by the specific Data ONTAP version.

NetApp Interoperability
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray).
You use the Component Explorer to select the components and Data ONTAP version to refine
your search. You can click Show Results to display the list of supported configurations that
match the criteria.

NetApp Hardware Universe

About this task

You must coordinate with the storage array administrator to perform this task on the storage array.

Steps

1. Create LUNs on the storage array depending on the number of nodes in the MetroCluster
configuration.
Each node in the MetroCluster configuration requires array LUNs for the root aggregate, data
aggregate, and spares.
158 | MetroCluster Installation and Configuration Guide

2. Configure parameters on the storage array that are required to work with Data ONTAP.

FlexArray Virtualization Implementation Guide for Third-Party Storage

FlexArray Virtualization Implementation Guide for NetApp E-Series Storage

Switch ports required for a MetroCluster configuration with array LUNs

When you are setting up a MetroCluster configuration with array LUNs, you must ensure that the
FC-VI and HBA ports from the controllers are connected to specific FC switch ports.

Switch ports required for a two-node MetroCluster configuration with array LUNs
When you are connecting Data ONTAP systems to FC switches for setting up a two-node
MetroCluster configuration with array LUNs, you must connect FC-VI and HBA ports from each
controller to specific switch ports.
If you are using both array LUNs and disks in the MetroCluster configuration, you must ensure that
the controller ports are connected to the switch ports recommended for configuration with disks, and
then use the remaining ports for configuration with array LUNs.
The following table lists the specific FC switch ports to which you must connect the different
controller ports in a two-node MetroCluster configuration with array LUNs:

FC_switch_x_1 FC_switch_x_2
Controller and Brocade Brocade Cisco 9148 Brocade Brocade Cisco 9148
port 6505 6510 or 9148s 6505 6510 or 9148s
controller_x 0 0 1 - - -
FC-VI port a
controller_x - - - 0 0 1
FC-VI port b
controller_x 1 1 2 - - -
HBA port a
controller_x - - - 1 1 2
HBA port b
controller_x 2 2 3 - - -
HBA port c
controller_x - - - 2 2 3
HBA port d

Note: The connections between the FC-VI ports and the FC switch ports are applicable only in the
case of a fabric-attached MetroCluster configuration.

Switch ports required for a four-node MetroCluster configuration with array LUNs
When you are connecting Data ONTAP systems to FC switches for setting up a four-node
MetroCluster configuration with array LUNs, you must connect FC-VI and HBA ports from each
controller to specific switch ports.
If you are using both array LUNs and disks in the MetroCluster configuration, you must ensure that
the controller ports are connected to the switch ports recommended for configuration with disks, and
then use the remaining ports for configuration with array LUNs.
The following table lists the specific FC switch ports to which you must connect the different
controller ports in a four-node MetroCluster configuration with array LUNs:
 Planning and installing a MetroCluster configuration with array LUNs | 159

FC_switch_x_1 FC_switch_x_2
Controller and Brocade Brocade Cisco 9148 Brocade Brocade Cisco 9148
port 6505 6510 or 9148s 6505 6510 or 9148s
controller_x_1 0 0 1 - - -
FC-VI port a
controller_x_1 - - - 0 0 1
FC-VI port b
controller_x_1 1 1 2 - - -
HBA port a
controller_x_1 - - - 1 1 2
HBA port b
controller_x_1 2 2 3 - - -
HBA port c
controller_x_1 - - - 2 2 3
HBA port d
controller_x_2 3 3 4 - - -
FC-VI port a
controller_x_2 - - - 3 3 4
FC-VI port b
controller_x_2 4 4 5 - - -
HBA port a
controller_x_2 - - - 4 4 5
HBA port b
controller_x_2 5 5 6 - - -
HBA port c
controller_x_2 - - - 5 5 6
HBA port d

Cabling the FC-VI and HBA ports in a MetroCluster configuration with array
LUNs
For a MetroCluster configuration with array LUNs, you must cable the FC-VI ports and the HBA
ports from the controllers based on the type of configuration.

About this task

In a stretch configuration, you must cable the FC-VI ports across controllers and the HBA ports to
the FC switches; in a fabric-attached configuration, you must cable the FC-VI ports and the HBA
ports to the FC switches.

You must connect the controllers in a MetroCluster configuration to the storage arrays through FC
switches. Direct connectivity between the controllers and storage arrays is not supported.

Choices
Cabling the FC-VI and HBA ports in a stretch MetroCluster configuration with array LUNs on
page 160
Cabling the FC-VI and HBA ports in a two-node fabric-attached MetroCluster configuration with
array LUNs on page 161
160 | MetroCluster Installation and Configuration Guide

Cabling the FC-VI and HBA ports in a four-node fabric-attached MetroCluster configuration with
array LUNs on page 162

Cabling the FC-VI and HBA ports in a stretch MetroCluster configuration with array LUNs
If you are setting up a two-node stretch MetroCluster configuration with array LUNs, you must cable
the FC-VI ports for direct connectivity between the controllers. In addition, you must cable each
controller's HBA ports to switch ports on the corresponding FC switches.

About this task

If you plan to use disks in addition to array LUNs in your MetroCluster configuration, you must use
the HBA ports and switch ports specified for the configuration with disks.
Recommended port assignments for FC switches on page 38

Steps

1. Cable the FC-VI ports on Controller A to the corresponding ports on Controller B.

2. Perform the controller-to-switch cabling at both the MetroCluster sites.

You must ensure redundancy in connections from the controller to the switches. Therefore, for
each controller, ensure that both the HBA ports in the same port pair are connected to alternate
FC switches.

Example
The following example shows the connections between HBA ports on Controller A and ports on
FC_switch_A_1 and FC_switch_A_2:

The following table lists the connections between the HBA ports and the FC switch ports in the
example illustration:

HBA ports Switch ports

Port pair
Port a FC_switch_A_1, Port 2
 Planning and installing a MetroCluster configuration with array LUNs | 161

HBA ports Switch ports

Port d FC_switch_A_2, Port 3
Port pair
Port b FC_switch_A_2, Port 2
Port c FC_switch_A_1, Port 3

After you finish

Cable the ISLs between the FC switches.

Related references
Switch ports required for a two-node MetroCluster configuration with array LUNs on page 158

Cabling the FC-VI and HBA ports in a two-node fabric-attached MetroCluster configuration
with array LUNs
If you are setting up a two-node fabric-attached MetroCluster configuration with array LUNs, you
must cable the FC-VI ports and the HBA ports to the switch ports.

About this task

You must repeat this task for each controller at both the MetroCluster sites.

If you plan to use disks in addition to array LUNs in your MetroCluster configuration, you must
use HBA ports and switch ports specified for configuration with disks.
Recommended port assignments for FC switches on page 38

Steps

1. Cable the FC-VI ports from the controller to alternate switch ports.

Example
The following example shows two FC-VI ports from Controller A cabled to switch ports on
alternate switches FC_switch_A_1 and FC_switch_A_2:

2. Perform the controller-to-switch cabling at both the MetroCluster sites.

162 | MetroCluster Installation and Configuration Guide

You must ensure redundancy in connections from the controller to the switches. Therefore, for
each controller at a site, ensure that both the HBA ports in the same port pair are connected to
alternate FC switches.

Example
The following example shows the connections between HBA ports on Controller A and ports on
FC_switch_A_1 and FC_switch_A_2:

The following table lists the connections between the HBA ports and the FC switch ports in the
illustration:

HBA ports Switch ports

Port pair
Port a FC_switch_A_1, Port 2
Port d FC_switch_A_2, Port 3
Port pair
Port b FC_switch_A_2, Port 2
Port c FC_switch_A_1, Port 3

After you finish

Cable the ISLs between the FC switches across the MetroCluster sites.

Related references
Switch ports required for a two-node MetroCluster configuration with array LUNs on page 158

Cabling the FC-VI and HBA ports in a four-node fabric-attached MetroCluster configuration
with array LUNs
If you are setting up a four-node fabric-attached MetroCluster configuration with array LUNs, you
must cable the FC-VI ports and the HBA ports to the switch ports.

About this task

You must repeat this task for each controller at both the MetroCluster sites.

If you plan to use disks in addition to array LUNs in your MetroCluster configuration, you must
use HBA ports and switch ports specified for configuration with disks.
Recommended port assignments for FC switches on page 38
 Planning and installing a MetroCluster configuration with array LUNs | 163

Steps

1. Cable the FC-VI ports from each controller to ports on alternate FC switches.

Example
The following example shows the connections between FC-VI ports and switch ports at Site A:

2. Perform the controller-to-switch cabling at both the MetroCluster sites.

You must ensure redundancy in connections from the controller to the switches. Therefore, for
each controller at a site, ensure that both the HBA ports in the same port pair are connected to
alternate FC switches.

Example
The following example shows the connections between HBA ports and switch ports at Site A:

The following table lists the connections between the HBA ports on controller_A_1 and the FC
switch ports in the illustration:
164 | MetroCluster Installation and Configuration Guide

HBA ports Switch ports

Port pair
Port a FC_switch_A_1, Port 2
Port d FC_switch_A_2, Port 3
Port pair
Port b FC_switch_A_2, Port 2
Port c FC_switch_A_1, Port 3

The following table lists the connections between the HBA ports on controller_A_2 and the FC
switch ports in the illustration:

HBA ports Switch ports

Port pair
Port a FC_switch_A_1, Port 5
Port d FC_switch_A_2, Port 6
Port pair
Port b FC_switch_A_2, Port 5
Port c FC_switch_A_1, Port 6

After you finish

Cable the ISLs between the FC switches across the MetroCluster sites.

Related references
Switch ports required for a four-node MetroCluster configuration with array LUNs on page 158

Cabling the ISLs between MetroCluster sites

You must connect the FC switches at each site through the fiber-optic Inter-Switch Links (ISLs) to
form the switch fabrics that connect the MetroCluster components.

About this task

This task includes steps performed at each MetroCluster site.
Up to four ISLs per FC switch fabric are supported.

Step

1. Connect the FC switches at each site to the ISL or ISLs.

This must be done for both switch fabrics.
Note: The Brocade and Cisco switches use different port numbering:

On Brocade switches, the first port is numbered 0.

On Cisco switches, the first port is numbered 1.

The port numbering is shown in the following table:

 Planning and installing a MetroCluster configuration with array LUNs | 165

ISL Site A Site B

To this port on FC_switch_A_x... To this port on FC_switch_B_x...
Brocade Brocade 6510 Cisco Brocade Brocade Cisco
6505 9148 or 6505 6510 9148 or
Cisco Cisco
9148S 9148S
First ISL 8 20 36 8 20 36
Second ISL 9 21 40 9 21 40
Third ISL 10 22 44 10 22 44
Four ISL 11 23 48 11 23 48

Related concepts
Port assignments for FC switches in a four-node configuration on page 38
Port assignments for FC switches in a two-node configuration on page 40

Cabling the cluster interconnect in four-node configurations

In a four-node MetroCluster configuration, you must cable the cluster interconnect between the local
controllers at each site. If the local controllers are not configured as two-node switchless clusters,
two-cluster switches are required.

About this task

This task is not required on two-node MetroCluster configurations.
This task must be performed at both MetroCluster sites.

Step

1. Cable the cluster interconnect from one controller to the other, or, if cluster interconnect switches
are used, from each controller to the switches.

If the storage Do this if using cluster Do this if using a switchless

controller is a... interconnect switches... cluster...
80xx Connect ports e0a through e0d on Connect ports e0a through e0d on
each controller to a local cluster each controller to the same ports on
interconnect switch. its HA partner.
62xx Connect ports e0c and e0e on each Connect ports e0c and e0e on each
controller to a local cluster controller to the same ports on its HA
interconnect switch. partner.
32xx Connect ports e1a and e2a on each Connect ports e1a and e2a on each
controller to a local cluster controller to the same ports on its HA
interconnect switch. partner.

Related information
Clustered Data ONTAP 8.3 Network Management Guide
166 | MetroCluster Installation and Configuration Guide

Cabling the cluster peering connections

You must ensure that the controller ports used for cluster peering have connectivity with the cluster
on the partner site.

About this task

This task must be performed on each controller in the MetroCluster configuration.
At least two ports on each controller should be used for cluster peering.
The recommended minimum bandwidth for the ports and network connectivity is 1 GbE.

Step

1. Identify and cable at least two ports for cluster peering and ensure they have network connectivity
with the partner cluster.
Cluster peering can be done on dedicated ports or on data ports. Using dedicated ports ensures
higher throughput for the cluster peering traffic.

Related concepts
Considerations for configuring cluster peering on page 12

Related information
Clustered Data ONTAP 8.3 Data Protection Guide
Clustered Data ONTAP 8.3 Cluster Peering Express Guide

Cabling the HA interconnect, if necessary

If you have a four-node MetroCluster configuration and the storage controllers in the HA pair are in
separate chassis, you must cable the HA interconnect between the controllers.

About this task

This task does not apply to two-node MetroCluster configurations.

This task must be performed at both MetroCluster sites.

The HA interconnect must be cabled only if the storage controllers in the HA pair are in separate
chassis.
Some storage controller models support two controllers in a single chassis, in which case they use
an internal HA interconnect.

Steps

1. Cable the HA interconnect if the storage controller's HA partner is in a separate chassis.

If the storage controller is Do this...

a...
80xx with I/O Expansion
a. Connect port ib0a on the first controller in the HA pair to port ib0a on
Module
the other controller.

b. Connect port ib0b on the first controller in the HA pair to port ib0b
on the other controller.
 Planning and installing a MetroCluster configuration with array LUNs | 167

If the storage controller is Do this...

a...
62xx
a. Connect port 2a (top port of NVRAM8 card in vertical slot 2) on the
first controller in the HA pair to port 2a on the other controller.

b. Connect port 2b (bottom port of NVRAM8 card in vertical slot 2) on

the first controller in the HA pair to port 2b on the other controller.

32xx
a. Connect port c0a on the first controller in the HA pair to port c0a on
the other controller.

b. Connect port c0b on the first controller in the HA pair to port c0b on
the other controller.

2. Repeat this task at the MetroCluster partner site.

Related information
Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems

Cabling the management and data connections

You must cable the management and data ports on each storage controller to the site networks.

About this task

This task must be repeated for each controller at both MetroCluster sites.
You can connect the controller and cluster switch management ports to existing switches in your
network or to new dedicated network switches such as NetApp CN1601 cluster management
switches.

Step

1. Cable the controller's management and data ports to the management and data networks at the
local site.
Installation and Setup Instructions FAS8040/FAS8060 Systems
Installation and setup Instructions FAS80xx Systems with I/O Expansion Modules
Installation and Setup Instructions FAS8020 systems
Installation and Setup Instructions 62xx Systems
Installation and Setup Instructions 32xx Systems

Cabling storage arrays to FC switches in a MetroCluster configuration

You must connect storage arrays to FC switches so that the Data ONTAP systems in the MetroCluster
configuration can access a specific array LUN through at least two paths.

Before you begin

The storage arrays must be set up to present array LUNs to Data ONTAP.
168 | MetroCluster Installation and Configuration Guide

The Data ONTAP controllers must be connected to the FC switches.

The ISLs must be cabled between the FC switches across the MetroCluster sites.

About this task

You must repeat this task for each storage array at both the MetroCluster sites.

You must connect the controllers in a MetroCluster configuration to the storage arrays through FC
switches. Direct connectivity between the controllers and storage arrays is not supported.

Step

1. Connect the storage array ports to FC switch ports.

At each site, connect the redundant port pairs in the storage array to FC switches on alternate
fabrics. This ensures redundancy in the paths for accessing the array LUNs.

Examples of cabling storage arrays to FC switches

Example of cabling storage array ports to FC switches in a four-node MetroCluster
configuration on page 169
Example of cabling storage array ports to FC switches in a two-node MetroCluster
configuration on page 168

Related concepts
Switch zoning in a MetroCluster configuration with array LUNs on page 171

Example of cabling storage array ports to FC switches in a two-node MetroCluster

configuration
In a MetroCluster configuration with array LUNs, you must connect the storage array ports that form
a redundant port pair to alternate FC switches.
The following illustration shows the connections between storage arrays and FC switches in a two-
node stretch MetroCluster configuration with array LUNs:

The connections between storage array ports and FC switch ports are similar for both stretch and
fabric-attached variants of two-node MetroCluster configurations with array LUNs.
 Planning and installing a MetroCluster configuration with array LUNs | 169

Note: If you plan to use disks in addition to array LUNs in your MetroCluster configuration, you
must use the switch ports specified for the configuration with disks.
Recommended port assignments for FC switches on page 38

In the illustration, the redundant array port pairs for both the sites are as follows:

Storage array at Site A:

Ports 1A and 2A

Ports 1B and 2B
Storage array at Site B:

Ports 1A' and 2A'

Ports 1B' and 2B'

FC_switch_A_1 at Site A and FC_switch_B_1 at Site B are connected to form fabric_1. Similarly,
FC_switch_A_2 at Site A and FC_switch_B_2 are connected to form fabric_2.
The following table lists the connections between the storage array ports and the FC switches for the
example MetroCluster illustration:

Array LUN ports FC switch ports Switch fabrics

Site A
1A FC_switch_A_1, Port 9 fabric_1
2A FC_switch_A_2, Port 10 fabric_2
1B FC_switch_A_1, Port 10 fabric_1
2B FC_switch_A_2, Port 9 fabric_2
Site B
1A' FC_switch_B_1, Port 9 fabric_1
2A' FC_switch_B_2, Port 10 fabric_2
1B' FC_switch_B_1, Port 10 fabric_1
2B' FC_switch_B_2, Port 9 fabric_2

Example of cabling storage array ports to FC switches in a four-node MetroCluster

configuration
In a MetroCluster configuration with array LUNs, you must connect the storage array ports that form
a redundant port pair to alternate FC switches.
The following reference illustration shows the connections between storage arrays and FC switches
in a four-node MetroCluster configuration with array LUNs:
170 | MetroCluster Installation and Configuration Guide

Note: If you plan to use disks in addition to array LUNs in your MetroCluster configuration, you
must use the switch ports specified for the configuration with disks.
Recommended port assignments for FC switches on page 38

In the illustration, the redundant array port pairs for both the sites are as follows:
Storage array at Site A:

Ports 1A and 2A

Ports 1B and 2B

Ports 1C and 2C

Ports 1D and 2D

Storage array at Site B:

Ports 1A' and 2A'

Ports 1B' and 2B'

Ports 1C' and 2C'

Ports 1D' and 2D'

FC_switch_A_1 at Site A and FC_switch_B_1 at Site B are connected to form fabric_1. Similarly,
FC_switch_A_2 at Site A and FC_switch_B_2 are connected to form fabric_2.
The following table lists the connections between the storage array ports and the FC switches for the
MetroCluster illustration:

Array LUN ports FC switch ports Switch fabrics

Site A
1A FC_switch_A_1, Port 7 fabric_1
2A FC_switch_A_2, Port 11 fabric_2
1B FC_switch_A_1, Port 8 fabric_1
2B FC_switch_A_2, Port 10 fabric_2
1C FC_switch_A_1, Port 9 fabric_1
2C FC_switch_A_2, Port 9 fabric_2
 Planning and installing a MetroCluster configuration with array LUNs | 171

Array LUN ports FC switch ports Switch fabrics

1D FC_switch_A_1, Port 10 fabric_1
2D FC_switch_A_2, Port 8 fabric_2
Site B
1A' FC_switch_B_1, Port 7 fabric_1
2A' FC_switch_B_2, Port 11 fabric_2
1B' FC_switch_B_1, Port 8 fabric_1
2B' FC_switch_B_2, Port 10 fabric_2
1C' FC_switch_B_1, Port 9 fabric_1
2C' FC_switch_B_2, Port 9 fabric_2
1D' FC_switch_B_1, Port 10 fabric_1
2D' FC_switch_B_2, Port 8 fabric_2

Switch zoning in a MetroCluster configuration with array

LUNs
Configuring switch zoning enables you to define which array LUNs can be viewed by a specific Data
ONTAP system in the MetroCluster configuration.

Related concepts
Example of switch zoning in a two-node MetroCluster configuration with array LUNs on page
172
Example of switch zoning in a four-node MetroCluster configuration with array LUNs on page
173

Requirements for switch zoning in a MetroCluster configuration with array

LUNs
When using switch zoning in a MetroCluster configuration with array LUNs, you must ensure that
certain basic requirements are followed.
The requirements for switch zoning in a MetroCluster configuration with array LUNs are as follows:

The MetroCluster configuration must follow the single-initiator to single-target zoning scheme.
Single-initiator to single-target zoning limits each zone to a single FC initiator port and a single
target port.

The FC-VI ports must be zoned end-to-end across the fabric.

Sharing of multiple initiator ports with a single target port is not supported in MetroCluster
configurations.
Similarly, sharing of multiple target ports with a single initiator port is also not supported.

You must have performed a basic configuration of the FC switches used in the MetroCluster
configuration.
Configuring the Cisco or Brocade FC switches manually on page 50
172 | MetroCluster Installation and Configuration Guide

Example of switch zoning in a two-node MetroCluster configuration with

array LUNs
Switch zoning defines paths between connected nodes. Configuring the zoning enables you to define
which array LUNs can be viewed by a specific Data ONTAP system.
You can use the following example as a reference when determining zoning for a two-node fabric-
attached MetroCluster configuration with array LUNs:

The following illustration shows zoning for a two-node stretch MetroCluster configuration with array
LUNs:

The examples show single-initiator to single-target zoning for the MetroCluster configurations. The
lines in the examples represent zones rather than connections; each line is labeled with its zone
number.
In the two illustrations, array LUNs are allocated on each storage array. LUNs of equal size are
provisioned on the storage arrays at both sites, which is a SyncMirror requirement. Each Data
ONTAP system has two paths to array LUNs. The ports on the storage array are redundant.
The redundant array port pairs for both the sites are as follows:

Storage array at Site A:

Ports 1A and 2A
 Planning and installing a MetroCluster configuration with array LUNs | 173

Ports 1B and 2B

Storage array at Site B:

Ports 1A' and 2A'

Ports 1B' and 2B'

The redundant port pairs on each storage array form alternate paths. Therefore, both the ports of the
port pairs can access the LUNs on the respective storage arrays.
The following table shows the zones for the illustrations:

Zone Data ONTAP controller and initiator port Storage array port
FC_switch_A_1
z1 Controller A: Port 0a Port 1A
z3 Controller A: Port 0c Port 1A'
FC_switch_A_2
z2 Controller A: Port 0b Port 2A'
z4 Controller A: Port 0d Port 2A
FC_switch_B_1
z5 Controller B: Port 0a Port 1B'
z7 Controller B: Port 0c Port 1B
FC_switch_B_2
z6 Controller B: Port 0b Port 2B
z8 Controller B: Port 0d Port 2B'

The following table shows the zones for the FC-VI connections in the fabric-attached configuration:

Zone Data ONTAP controller and initiator port Switch

Site A
zX Controller A: Port FC-VI a FC_switch_A_1
zY Controller A: Port FC-VI b FC_switch_A_2
Site B
zX Controller B: Port FC-VI a FC_switch_B_1
zY Controller B: Port FC-VI b FC_switch_B_2

Example of switch zoning in a four-node MetroCluster configuration with

array LUNs
Switch zoning defines paths between connected nodes. Configuring the zoning enables you to define
which array LUNs can be viewed by a specific Data ONTAP system.
You can use the following example as a reference when determining zoning for a four-node
MetroCluster configuration with array LUNs. The example shows single-initiator to single-target
zoning for a MetroCluster configuration. The lines in the following example represent zones rather
than connections; each line is labeled with its zone number:
174 | MetroCluster Installation and Configuration Guide

In the illustration, array LUNs are allocated on each storage array for the MetroCluster configuration.
LUNs of equal size are provisioned on the storage arrays at both sites, which is a SyncMirror
requirement. Each Data ONTAP system has two paths to array LUNs. The ports on the storage array
are redundant.
In the illustration, the redundant array port pairs for both the sites are as follows:
Storage array at Site A:

Ports 1A and 2A

Ports 1B and 2B

Ports 1C and 2C

Ports 1D and 2D

Storage array at Site B:

Ports 1A' and 2A'

Ports 1B' and 2B'

Ports 1C' and 2C'

Ports 1D' and 2D'

The redundant port pairs on each storage array form alternate paths. Therefore, both the ports of the
port pairs can access the LUNs on the respective storage arrays.
The following table shows the zones for this example:

Zone Data ONTAP controller and initiator port Storage array port
FC_switch_A_1
z1 Controller_A_1: Port 0a Port 1A
z3 Controller_A_1: Port 0c Port 1A'
z5 Controller_A_2: Port 0a Port 1B
z7 Controller_A_2: Port 0c Port 1B'
FC_switch_A_2
z2 Controller_A_1: Port 0b Port 2A'
z4 Controller_A_1: Port 0d Port 2A
 Planning and installing a MetroCluster configuration with array LUNs | 175

Zone Data ONTAP controller and initiator port Storage array port
z6 Controller_A_2: Port 0b Port 2B'
z8 Controller_A_2: Port 0d Port 2B
FC_switch_B_1
z9 Controller_B_1: Port 0a Port 1C'
z11 Controller_B_1: Port 0c Port 1C
z13 Controller_B_2: Port 0a Port 1D'
z15 Controller_B_2: Port 0c Port 1D
FC_switch_B_2
z10 Controller_B_1: Port 0b Port 2C
z12 Controller_B_1: Port 0d Port 2C'
z14 Controller_B_2: Port 0b Port 2D
z16 Controller_B_2: Port 0d Port 2D'

The following table shows the zones for the FC-VI connections at Site A and Site B:

Zone Data ONTAP controller and FC initiator Switch

port
Site A
zX Controller_A_1: Port FC-VI a FC_switch_A_1
zY Controller_A_1: Port FC-VI b FC_switch_A_2
zX Controller_A_2: Port FC-VI a FC_switch_A_1
zY Controller_A_2: Port FC-VI b FC_switch_A_2
Site B
zX Controller_B_1: Port FC-VI a FC_switch_B_1
zY Controller_B_1: Port FC-VI b FC_switch_B_2
zX Controller_B_2: Port FC-VI a FC_switch_B_1
zY Controller_B_2: Port FC-VI b FC_switch_B_2

Setting up Data ONTAP in a MetroCluster configuration with

array LUNs
After connecting the devices in the MetroCluster configuration, you must set up the Data ONTAP
systems to use the storage on the storage array. You must also configure any required Data ONTAP
feature.

Steps
1. Verifying the HA state of components is mcc or mcc-2n in Maintenance mode on page 176
2. Configuring Data ONTAP on a system that uses only array LUNs on page 177
3. Setting up the cluster on page 179
4. Installing the license for using array LUNs in a MetroCluster configuration on page 180
176 | MetroCluster Installation and Configuration Guide

5. Assigning ownership of array LUNs on page 180

6. Peering the clusters on page 181
7. Mirroring the root aggregates on page 183
8. Creating a mirrored data aggregate on each node on page 184
9. Implementing the MetroCluster configuration on page 185
10. Configuring the MetroCluster FC switches for health monitoring on page 187
11. Checking the MetroCluster configuration on page 188

Verifying the HA state of components is mcc or mcc-2n in Maintenance

mode
In a MetroCluster configuration, you must set the HA state of the controller and chassis components
to mcc or mcc-2n so they boot up properly. For systems received from the factory, this value is pre-
configured and you do not need to verify it.

Before you begin

The system must be in Maintenance mode.

About this task

This task is not required on systems received from the factory.
This task must be performed on each node.

Steps

1. In Maintenance mode, display the HA state of the controller module and chassis:
ha-config show

The correct HA state depends on whether you have a four-node or two-node MetroCluster
configuration:

Number of controllers in the MetroCluster HA state for all components should be...
configuration
Four mcc
Two mcc-2n

2. If the displayed system state of the controller is not mcc, set the HA state for the controller
module to mcc or mcc-2n:

Number of controllers in Command

the MetroCluster
configuration
Four ha-config modify controller mcc
Two ha-config modify controller mcc-2n

3. If the displayed system state of the chassis is not mcc, set the HA state for the chassis to mcc or
mcc-2n:

Number of controllers in Command

the MetroCluster
configuration
Four ha-config modify chassis mcc
Two ha-config modify chassis mcc-2n
 Planning and installing a MetroCluster configuration with array LUNs | 177

4. Boot the node to Data ONTAP:

boot_ontap

5. Repeat these steps on each node in the MetroCluster configuration.

Configuring Data ONTAP on a system that uses only array LUNs

If you want to configure Data ONTAP for use with array LUNs, you must configure the root
aggregate and root volume, reserve space for diagnostics and recovery operations, and set up the
cluster.

Before you begin

The Data ONTAP system must be connected to the storage array.

The storage array administrator must have created LUNs and presented them to Data ONTAP.

The storage array administrator must have configured LUN security.

About this task

You must configure each node that you want to use with array LUNs. If the node is in an HA pair,
then you should complete the configuration process on one node before proceeding with the
configuration on the partner node.

Steps

1. Power on the primary node and interrupt the boot process by pressing Ctrl-C when you see the
following message on the console:
Press CTRL-C for special boot menu
.

2. Select option 4 (Clean configuration and initialize all disks) on the boot menu.
The list of array LUNs made available to Data ONTAP is displayed. In addition, the array LUN
size required for root volume creation is also specified. The size required for root volume creation
differs from one Data ONTAP system to another.

Example

If no array LUNs were previously assigned, Data ONTAP detects and displays the available
array LUNs, as shown in the following example:

**********************************************************************
* No disks or array LUNs are owned by this node. *
* You can use the following information to verify connectivity from *
* HBAs to switch ports. If the connectivity of HBAs to switch ports *
* does not match your expectations, configure your SAN and rescan. *
* You can rescan by entering 'r' at the prompt for selecting *
* array LUNs below. *
**********************************************************************

HBA HBA WWPN Switch port Switch port WWPN

--- -------- ----------- ----------------
v0 0a9032f400000000 switch0:0 9408b11e15000010
v1 0b9032f400000000 switch0:2 9408b11e15000210
v2 0c9032f400000000 switch1:0 9508b11e15000010
v3 0d9032f400000000 switch1:2 9508b11e15000210

No native disks were detected, but array LUNs were detected.

You will need to select an array LUN to be used to create the root aggregate and root volume.

The array LUNs visible to the system are listed below. Select one array LUN to be used to
create the root aggregate and root volume. The root volume requires 45.0 GB of space.

Warning: The contents of the array LUN you select will be erased by Data ONTAP prior to their use.

Index Array LUN Name Model Vendor Size Owner Checksum Serial Number
----- ----------------- ---------- -------- -------- ------ -------- ------------------------
0 switch0:5.183L1 SYMMETRIX EMC 266.1 GB Block 600604803436313734316631
1 switch0:5.183L3 SYMMETRIX EMC 266.1 GB Block 600604803436316333353837
2 switch0:5.183L31 SYMMETRIX EMC 266.1 GB Block 600604803436313237643666
178 | MetroCluster Installation and Configuration Guide

3 switch0:5.183L33 SYMMETRIX EMC 658.3 GB Block 600604803436316263613066

4 switch0:7.183L0 SYMMETRIX EMC 173.6 GB Block 600604803436313261356235
5 switch0:7.183L2 SYMMETRIX EMC 173.6 GB Block 600604803436313438396431
6 switch0:7.183L4 SYMMETRIX EMC 658.3 GB Block 600604803436313161663031
7 switch0:7.183L30 SYMMETRIX EMC 173.6 GB Block 600604803436316538353834
8 switch0:7.183L32 SYMMETRIX EMC 266.1 GB Block 600604803436313237353738
9 switch0:7.183L34 SYMMETRIX EMC 658.3 GB Block 600604803436313737333662

If array LUNs were previously assigned, for example, through the maintenance mode, they are
either marked local or partner in the list of the available array LUNs, depending on
whether the array LUNs were selected from the node on which you are installing Data
ONTAP or its HA partner.

**********************************************************************
* No disks are owned by this node, but array LUNs are assigned. *
* You can use the following information to verify connectivity from *
* HBAs to switch ports. If the connectivity of HBAs to switch ports *
* does not match your expectations, configure your SAN and rescan. *
* You can rescan by entering 'r' at the prompt for selecting *
* array LUNs below. *

**********************************************************************
HBA HBA WWPN Switch port Switch port WWPN
--- -------- ----------- ----------------
0e 500a098001baf8e0 vgbr6510s203:25 20190027f88948dd
0f 500a098101baf8e0 vgci9710s202:1-17 2011547feeead680
0g 500a098201baf8e0 vgbr6510s203:27 201b0027f88948dd
0h 500a098301baf8e0 vgci9710s202:1-18 2012547feeead680

No native disks were detected, but array LUNs were detected.

You will need to select an array LUN to be used to create the root aggregate and root volume.

The array LUNs visible to the system are listed below. Select one array LUN to be used to
create the root aggregate and root volume. The root volume requires 350.0 GB of space.

Warning: The contents of the array LUN you select will be erased by Data ONTAP prior to their use.

Index Array LUN Name Model Vendor Size Owner Checksum Serial Number
----- ----------------------- ------ ------ -------- ------ -------- ------------------------
0 vgci9710s202:2-24.0L19 RAID5 DGC 217.3 GB Block 6006016083402B0048E576D7
1 vgbr6510s203:30.126L20 RAID5 DGC 217.3 GB Block 6006016083402B0049E576D7
2 vgci9710s202:2-24.0L21 RAID5 DGC 217.3 GB Block 6006016083402B004AE576D7
3 vgbr6510s203:30.126L22 RAID5 DGC 405.4 GB local Block 6006016083402B004BE576D7
4 vgci9710s202:2-24.0L23 RAID5 DGC 217.3 GB Block 6006016083402B004CE576D7
5 vgbr6510s203:30.126L24 RAID5 DGC 217.3 GB Block 6006016083402B004DE576D7
6 vgbr6510s203:30.126L25 RAID5 DGC 423.5 GB local Block 6006016083402B003CF93694
7 vgci9710s202:2-24.0L26 RAID5 DGC 423.5 GB Block 6006016083402B003DF93694

In this example, array LUNs with index numbers 3 and 6 are marked local, because they had
been previously assigned from this particular node.

3. Select the index number corresponding to the array LUN you want to assign as the root volume.
Ensure that you select an array LUN of sufficient size to create the root volume.
The array LUN selected for root volume creation is marked local (root).

Example
In the following example, the array LUN with index number 3 is marked for root volume
creation.

The root volume will be created on switch 0:5.183L33.

Data ONTAP requires that 11.0 GB of space be reserved for use in diagnostic and recovery
operations. Select one array LUN to be used as spare for diagnostic and recovery operations.

Index Array LUN Name Model Vendor Size Owner Checksum Serial Number
----- ----------------- ---------- ------ -------- -------------- -------- ------------------------
0 switch0:5.183L1 SYMMETRIX EMC 266.1 GB Block 600604803436313734316631
1 switch0:5.183L3 SYMMETRIX EMC 266.1 GB Block 600604803436316333353837
2 switch0:5.183L31 SYMMETRIX EMC 266.1 GB Block 600604803436313237643666
3 switch0:5.183L33 SYMMETRIX EMC 658.3 GB local (root) Block 600604803436316263613066
4 switch0:7.183L0 SYMMETRIX EMC 173.6 GB Block 600604803436313261356235
5 switch0:7.183L2 SYMMETRIX EMC 173.6 GB Block 600604803436313438396431
6 switch0:7.183L4 SYMMETRIX EMC 658.3 GB Block 600604803436313161663031
7 switch0:7.183L30 SYMMETRIX EMC 173.6 GB Block 600604803436316538353834
8 switch0:7.183L32 SYMMETRIX EMC 266.1 GB Block 600604803436313237353738
9 switch0:7.183L34 SYMMETRIX EMC 658.3 GB Block 600604803436313737333662

4. Select the index number corresponding to the array LUN you wish to assign for use in diagnostic
and recovery options.
 Planning and installing a MetroCluster configuration with array LUNs | 179

Ensure that you select an array LUN with sufficient size for use in diagnostic and recovery
options. If required, you can also select multiple array LUNs with combined size greater than or
equal to the specified size.
Note: To select multiple entries, you must enter the comma-separated values of all the index
numbers corresponding to the array LUNs you wish to select for diagnostic and recovery
options.

The following example shows a list of array LUNs selected for root volume creation and for
diagnostic and recovery options.

Here is a list of the selected array LUNs

Index Array LUN Name Model Vendor Size Owner Checksum Serial Number
----- ----------------- --------- ------ -------- ------------- -------- ------------------------
2 switch0:5.183L31 SYMMETRIX EMC 266.1 GB local Block 600604803436313237643666
3 switch0:5.183L33 SYMMETRIX EMC 658.3 GB local (root) Block 600604803436316263613066
4 switch0:7.183L0 SYMMETRIX EMC 173.6 GB local Block 600604803436313261356235
5 switch0:7.183L2 SYMMETRIX EMC 173.6 GB local Block 600604803436313438396431
Do you want to continue?

5. Enter y when prompted by the system to continue with the installation process.
The root aggregate and the root volume are created and the rest of the installation process
continues.

6. Enter the required details to create the node management interface.

Example
The following example shows the node management interface screen with a message confirming
the creation of the node management interface:
Welcome to node setup.

You can enter the following commands at any time:

"help" or "?" - if you want to have a question clarified,
"back" - if you want to change previously answered questions, and
"exit" or "quit" - if you want to quit the setup wizard.
Any changes you made before quitting will be saved.

To accept a default or omit a question, do not enter a value.

Enter the node management interface port [e0M]:

Enter the node management interface IP address: 192.0.2.66

Enter the node management interface netmask: 255.255.255.192

Enter the node management interface default gateway: 192.0.2.7
A node management interface on port e0M with IP address 192.0.2.66 has been created.

This node has its management address assigned and is ready for cluster setup.

After you finish

After configuring Data ONTAP on all the nodes that you want to use with array LUNs, you should
complete the cluster setup process.
Clustered Data ONTAP 8.3 Software Setup Guide

Related information
FlexArray Virtualization Installation Requirements and Reference Guide

Setting up the cluster

Setting up the cluster involves setting up each node, creating the cluster on the first node, and joining
any remaining nodes to the cluster.

Related information
Clustered Data ONTAP 8.3 Software Setup Guide
180 | MetroCluster Installation and Configuration Guide

Installing the license for using array LUNs in a MetroCluster configuration

You must install the V_StorageAttach license on each MetroCluster node that you want to use with
array LUNs. You cannot use array LUNs in an aggregate until the license is installed.

Before you begin

The cluster must be installed.

You must have the license key for the V_StorageAttach license.

About this task

You must ensure that you use a separate license key for each node on which you want to install the
V_StorageAttach license.

Steps

1. Use the system license add command to install the V_StorageAttach license.
Repeat this step for each cluster node on which you want to install the license.
Clustered Data ONTAP 8.3.1 man page: system license add - Add one or more licenses
2. Use the system license show command to verify whether the V_StorageAttach license is
installed on all the required nodes in a cluster.

Example
The following example output shows that the V_StorageAttach license is installed on the nodes of
cluster_A:

cluster_A::> system license show

Serial Number: nnnnnnnn
Owner: controller_A_1
Package Type Description Expiration
----------------- ------- --------------------- --------------------
V_StorageAttach license Virtual Attached Storage

Serial Number: llllllll

Owner: controller_A_2
Package Type Description Expiration
----------------- ------- --------------------- --------------------
V_StorageAttach license Virtual Attached Storage

Clustered Data ONTAP 8.3.1 man page: system license show - Display licenses

Assigning ownership of array LUNs

Array LUNs must be owned by a node before they can be added to an aggregate to be used as
storage.

Before you begin

Back-end configuration testing (testing of the connectivity and configuration of devices behind
the Data ONTAP systems) must be completed.

Array LUNs that you want to assign must be presented to the Data ONTAP systems.
 Planning and installing a MetroCluster configuration with array LUNs | 181

About this task

You can assign ownership of array LUNs that have the following characteristics:

They are unowned.

They have no storage array configuration errors, such as the following:

The array LUN is smaller than or larger than the size that Data ONTAP supports.

The LDEV is mapped on only one port.

The LDEV has inconsistent LUN IDs assigned to it.

The LUN is available on only one path.

Data ONTAP issues an error message if you try to assign ownership of an array LUN with back-end
configuration errors that would interfere with the Data ONTAP system and the storage array
operating together. You must fix such errors before you can proceed with array LUN assignment.
Data ONTAP alerts you if you try to assign an array LUN with a redundancy error: for example, all
paths to this array LUN are connected to the same controller or only one path to the array LUN. You
can fix a redundancy error before or after assigning ownership of the LUN.

Steps

1. Enter the following command to see the array LUNs that have not yet been assigned to a node:
storage disk show -container-type unassigned

2. Enter the following command to assign an array LUN to this node:

storage disk assign -disk arrayLUNname -owner nodename

If you want to fix a redundancy error after disk assignment instead of before, you must use the -
force parameter with the storage disk assign command.

Related information
FlexArray Virtualization Installation Requirements and Reference Guide

Peering the clusters

The clusters in the MetroCluster configuration must be in a peer relationship so that they can
communicate with each other and perform the data mirroring essential to MetroCluster disaster
recovery.

Before you begin

For systems received from the factory, the cluster peering is configured and you do not need to
manually peer the clusters.

Related concepts
Considerations when using dedicated ports on page 13
Considerations when sharing data ports on page 13

Related references
Prerequisites for cluster peering on page 12

Related information
Clustered Data ONTAP 8.3 Data Protection Guide
182 | MetroCluster Installation and Configuration Guide

Clustered Data ONTAP 8.3 Cluster Peering Express Guide

Configuring intercluster LIFs

You must create intercluster LIFs on ports used for communication between the MetroCluster partner
clusters. You can use dedicated ports or ports that also have data traffic.

Creating the cluster peer relationship

You create the cluster peer relationship using a set of intercluster logical interfaces to make
information about one cluster available to the other cluster for use in cluster peering applications.

Before you begin

Intercluster LIFs should be created in the IPspaces of both clusters you want to peer.

You should ensure that the intercluster LIFs of the clusters can route to each other.

If there are different administrators for each cluster, the passphrase used to authenticate the
cluster peer relationship should be agreed upon.

About this task

If you created intercluster LIFs in a nondefault IPspace, you need to designate the IPspace when you
create the cluster peer.

Steps

1. Create the cluster peer relationship on each cluster by using the cluster peer create
command.
The passphrase that you use is not displayed as you type it.
If you created a nondefault IPspace to designate intercluster connectivity, you use the ipspace
parameter to select that IPspace.

Example
In the following example, cluster01 is peered with a remote cluster named cluster02. Cluster01 is
a two-node cluster that has one intercluster LIF per node. The IP addresses of the intercluster
LIFs created in cluster01 are 192.168.2.201 and 192.168.2.202. Similarly, cluster02 is a two-node
cluster that has one intercluster LIF per node. The IP addresses of the intercluster LIFs created in
cluster02 are 192.168.2.203 and 192.168.2.204. These IP addresses are used to create the cluster
peer relationship.

cluster01::> cluster peer create -peer-addrs

192.168.2.203,192.168.2.204
Please type the passphrase:
Please type the passphrase again:

cluster02::> cluster peer create -peer-addrs

192.168.2.201,192.168.2.202
Please type the passphrase:
Please type the passphrase again:

If DNS is configured to resolve host names for the intercluster IP addresses, you can use host
names in the peer-addrs option. It is not likely that intercluster IP addresses frequently
change; however, using host names allows intercluster IP addresses to change without having to
modify the cluster peer relationship.
 Planning and installing a MetroCluster configuration with array LUNs | 183

Example
In the following example, an IPspace called IP01A was created on cluster01 for intercluster
connectivity. The IP addresses used in the previous example are used in this example to create the
cluster peer relationship.

cluster01::> cluster peer create -peer-addrs

192.168.2.203,192.168.2.204
-ipspace IP01A
Please type the passphrase:
Please type the passphrase again:

cluster02::> cluster peer create -peer-addrs

192.168.2.201,192.168.2.202
Please type the passphrase:
Please type the passphrase again:

2. Display the cluster peer relationship by using the cluster peer show command with the -
instance parameter.

Displaying the cluster peer relationship verifies that the relationship was established successfully.

Example

cluster01::> cluster peer show instance

Peer Cluster Name: cluster02
Remote Intercluster Addresses: 192.168.2.203,192.168.2.204
Availability: Available
Remote Cluster Name: cluster02
Active IP Addresses: 192.168.2.203,192.168.2.204
Cluster Serial Number: 1-80-000013

3. Preview the health of the nodes in the peer cluster by using the cluster peer health show
command.
Previewing the health checks the connectivity and status of the nodes on the peer cluster.

Example

cluster01::> cluster peer health show

Node cluster-Name Node-Name
Ping-Status RDB-Health Cluster-Health Avail
---------- --------------------------- --------- --------------- --------
cluster01-01
cluster02 cluster02-01
Data: interface_reachable
ICMP: interface_reachable true true true
cluster02-02
Data: interface_reachable
ICMP: interface_reachable true true true
cluster01-02
cluster02 cluster02-01
Data: interface_reachable
ICMP: interface_reachable true true true
cluster02-02
Data: interface_reachable
ICMP: interface_reachable true true true

Mirroring the root aggregates

You must mirror the root aggregates in your MetroCluster configuration to ensure data protection.

Before you begin

You must have ensured that the SyncMirror requirements for the MetroCluster configuration with
array LUNs are satisfied.
Requirements for a MetroCluster configuration with array LUNs on page 155
184 | MetroCluster Installation and Configuration Guide

About this task

You must repeat this task for each controller in the MetroCluster configuration.

Step

1. Use the storage aggregate mirror command to mirror the unmirrored root aggregate.

Example
The following command mirrors the root aggregate for controller_A_1:

controller_A_1::> storage aggregate mirror aggr0_controller_A_1

The root aggregate is mirrored with array LUNs from pool1.

Creating a mirrored data aggregate on each node

You must create a mirrored data aggregate on each node in the DR group.

Before you begin

You should know what drives or array LUNs will be used in the new aggregate.
If you have multiple drive types in your system (heterogeneous storage), you should understand
how you can ensure that the correct drive type is selected.

About this task

Drives and array LUNs are owned by a specific node; when you create an aggregate, all drives in
that aggregate must be owned by the same node, which becomes the home node for that
aggregate.

Aggregate names should conform to the naming scheme you determined when you planned your
MetroCluster configuration.

The Clustered Data ONTAP Data Protection Guide contains more information about mirroring
aggregates.

Steps

1. Display a list of available spares:

storage disk show -spare -owner node_name

2. Create the aggregate by using the storage aggregate create -mirror true command.
If you are logged in to the cluster on the cluster management interface, you can create an
aggregate on any node in the cluster. To ensure that the aggregate is created on a specific node,
use the -node parameter or specify drives that are owned by that node.
You can specify the following options:

Aggregate's home node (that is, the node that owns the aggregate in normal operation)

List of specific drives or array LUNs that are to be added to the aggregate

Number of drives to include

Checksum style to use for the aggregate

Type of drives to use

 Planning and installing a MetroCluster configuration with array LUNs | 185

Size of drives to use

Drive speed to use

RAID type for RAID groups on the aggregate

Maximum number of drives or array LUNs that can be included in a RAID group

Whether drives with different RPM are allowed

For more information about these options, see the storage aggregate create man page.

Example
The following command creates a mirrored aggregate with 10 disks:

controller_A_1::> storage aggregate create aggr1_controller_A_1 -

diskcount 10 -node controller_A_1 -mirror true
[Job 15] Job is queued: Create aggr1_controller_A_1.
[Job 15] The job is starting.
[Job 15] Job succeeded: DONE

3. Verify the RAID group and drives of your new aggregate:

storage aggregate show-status -aggregate aggregate-name

Implementing the MetroCluster configuration

You must run the metrocluster configure command to start data protection in the MetroCluster
configuration.

Before you begin

There must be at least two nonroot mirrored data aggregates on each cluster, and all aggregates
must be mirrored.
You can verify this with the storage aggregate show command.

The ha-config state of the controllers and chassis must be mcc.

This state is preconfigured on systems shipped from the factory.

About this task

You issue the metrocluster configure command once, on any of the nodes, to enable the
MetroCluster configuration. You do not need issue the command on each of the sites or nodes, and it
does not matter which node or site you choose to issue the command on.
The metrocluster configure command automatically pairs the two nodes with lowest system
IDs in each of the two clusters as DR partners. In a four-node MetroCluster, there are two DR partner
pairs. The second DR pair is created from the two nodes with higher system IDs.

Steps

1. Enter the metrocluster configure command in the following format, depending on whether
you .

If your MetroCluster Then take these steps:

configuration will have..
Multiple data aggregates From any node's prompt, perform the MetroCluster configuration
operation:
This is the best practice.
metrocluster configure node-name
186 | MetroCluster Installation and Configuration Guide

If your MetroCluster Then take these steps:

configuration will have..
A single data aggregate
a. From any node's prompt, change to the advanced privilege level:
set -privilege advanced
You need to respond with y when prompted to continue into
advanced mode and see the advanced mode prompt (*>).

b. Perform the MetroCluster configuration operation with the -allow-

with-one-aggregate true parameter:
metrocluster configure -allow-with-one-
aggregate true node-name

c. Return to the admin privilege level:

set -privilege admin

Example
The following command enables MetroCluster configuration on all nodes in the DR group that
contains controller_A_1:

controller_A_1::*> metrocluster configure -node-name controller_A_1

[Job 121] Job succeeded: Configure is successful.

2. Check the networking status on site A:

network port show

Example

cluster_A::> network port show

Speed (Mbps)
Node Port IPspace Broadcast Domain Link MTU Admin/Oper
------ --------- --------- ---------------- ----- ------- ------------
controller_A_1
e0a Cluster Cluster up 9000 auto/1000
e0b Cluster Cluster up 9000 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
e0e Default Default up 1500 auto/1000
e0f Default Default up 1500 auto/1000
e0g Default Default up 1500 auto/1000
controller_A_2
e0a Cluster Cluster up 9000 auto/1000
e0b Cluster Cluster up 9000 auto/1000
e0c Default Default up 1500 auto/1000
e0d Default Default up 1500 auto/1000
e0e Default Default up 1500 auto/1000
e0f Default Default up 1500 auto/1000
e0g Default Default up 1500 auto/1000
14 entries were displayed.

3. Confirm the MetroCluster configuration from both sites in the MetroCluster configuration:

a. Confirm the configuration from site A:

metrocluster show
 Planning and installing a MetroCluster configuration with array LUNs | 187

Example

cluster_A::> metrocluster show

Cluster Configuration State Mode
------------------------- ------------------- -----------
Local: cluster_A configured normal
Remote: cluster_B configured normal

b. Confirm the configuration from site B:

metrocluster show

Example

cluster_B::> metrocluster show

Cluster Configuration State Mode
------------------------- ------------------- -----------
Local: cluster_B configured normal
Remote: cluster_A configured normal

Configuring the MetroCluster FC switches for health monitoring

In a fabric-attached MetroCluster configuration, you must perform some special configuration steps
to monitor the FC switches.

Steps

1. Issue the following command on each MetroCluster node:

storage switch add -switch-ipaddress ipaddress

This command must be repeated on all four switches in the MetroCluster configuration.

Example
The following example shows the command to add a switch with IP address of 10.10.10.10:

controller_A_1::> storage switch add -switch-ipaddress 10.10.10.10

2. Verify that all switches are properly configured:

storage switch show

It might take up to 15 minutes to reflect all data due to the 15-minute polling interval.

Example
The following example shows the command given to verify the MetroCluster's FC switches are
configured:

controller_A_1::> storage switch show

Fabric Switch Name Vendor Model Switch WWN Status
---------------- --------------- ------- ------------ ---------------- ------
1000000533a9e7a6 brcd6505-fcs40 Brocade Brocade6505 1000000533a9e7a6 OK
1000000533a9e7a6 brcd6505-fcs42 Brocade Brocade6505 1000000533d3660a OK
1000000533ed94d1 brcd6510-fcs44 Brocade Brocade6510 1000000533eda031 OK
1000000533ed94d1 brcd6510-fcs45 Brocade Brocade6510 1000000533ed94d1 OK
4 entries were displayed.

controller_A_1::>

If the switch's worldwide name (WWN) is shown, the Data ONTAP health monitor is able to
contact and monitor the FC switch.
188 | MetroCluster Installation and Configuration Guide

Related information
Clustered Data ONTAP 8.3 System Administration Guide

Checking the MetroCluster configuration

You can check that the components and relationships in the MetroCluster configuration are working
correctly. You should do a check after initial configuration, and after making any changes to the
MetroCluster configuration, and before a negotiated (planned) switchover or a switchback operation.
After you run the metrocluster check run command, you then display the results of the check
with various metrocluster check show commands.

About this task

If the metrocluster check run command is issued twice within a short time, on either or both
clusters, a conflict can occur and the command might not collect all data. Subsequent
metrocluster check show commands will not show the expected output.

Steps

1. Check the configuration:

metrocluster check run

Example
The command will run as a background job:

controller_A_1::> metrocluster check run

[Job 60] Job succeeded: Check is successful. Run "metrocluster check
show" command to view the results of this operation.

controller_A_1::> metrocluster check show

Last Checked On: 5/22/2015 12:23:28

Component Result
------------------- ---------
nodes ok
lifs ok
config-replication ok
aggregates ok
clusters ok
5 entries were displayed.

2. Display more detailed results from the most recent metrocluster check run command:
metrocluster check aggregate show
metrocluster check cluster show
metrocluster check config-replication show
metrocluster check lif show
metrocluster check node show

The metrocluster check show commands show the results of the most recent
metrocluster check run command. You should always run the metrocluster check
run command prior to using the metrocluster check show commands to ensure that the
information displayed is current.
 Planning and installing a MetroCluster configuration with array LUNs | 189

Example
The following example shows the metrocluster check aggregate show output for a
healthy four-node MetroCluster configuration:

controller_A_1::> metrocluster check aggregate show

Last Checked On: 8/5/2014 00:42:58

Node Aggregate Check Result

--------------------- --------------------- --------------------- ---------
controller_A_1 aggr0_controller_A_1_0
mirroring-status ok
disk-pool-allocation ok
controller_A_1_aggr1
mirroring-status ok
disk-pool-allocation ok
controller_A_2 aggr0_controller_A_2
mirroring-status ok
disk-pool-allocation ok
controller_A_2_aggr1
mirroring-status ok
disk-pool-allocation ok
controller_B_1 aggr0_controller_B_1_0
mirroring-status ok
disk-pool-allocation ok
controller_B_1_aggr1
mirroring-status ok
disk-pool-allocation ok
controller_B_2 aggr0_controller_B_2
mirroring-status ok
disk-pool-allocation ok
controller_B_2_aggr1
mirroring-status ok
disk-pool-allocation ok
16 entries were displayed.

The following example shows the metrocluster check cluster show output for a healthy
four-node MetroCluster configuration. It indicates that the clusters are ready to perform a
negotiated switchover if necessary.

Last Checked On: 5/22/2015 12:23:28

Cluster Check Result

--------------------- ------------------------------------ ---------
cluster_A
negotiated-switchover-ready ok
switchback-ready not-
applicable
job-schedules ok
licenses ok
cluster_B
negotiated-switchover-ready ok
switchback-ready not-
applicable
job-schedules ok
licenses ok
8 entries were displayed.

Related information
Clustered Data ONTAP 8.3 Physical Storage Management Guide
Clustered Data ONTAP 8.3 Network Management Guide
Clustered Data ONTAP 8.3 Data Protection Guide
190 | MetroCluster Installation and Configuration Guide

Implementing a MetroCluster configuration with both disks

and array LUNs
To implement a MetroCluster configuration with native disks and array LUNs, you must ensure that
the Data ONTAP systems used in the configuration can attach to storage arrays.
A MetroCluster configuration with disks and array LUNs can have either two or four nodes.
Although the four-node MetroCluster configuration must be fabric-attached, the two-node
configuration can either be stretch or fabric-attached.
NetApp Interoperability Matrix Tool
After opening the Interoperability Matrix, you can use the Storage Solution field to select your
MetroCluster solution (fabric MetroCluster or stretch MetroCluster, with or without FlexArray). You
use the Component Explorer to select the components and Data ONTAP version to refine your
search. You can click Show Results to display the list of supported configurations that match the
criteria.

Related concepts
Example of a four-node MetroCluster configuration with disks and array LUNs on page 194

Related references
Examples of two-node stretch MetroCluster configurations with disks and array LUNs on page
191
Example of a two-node fabric-attached MetroCluster configuration with disks and array LUNs on
page 193

Considerations when implementing a MetroCluster configuration with disks

and array LUNs
When planning your MetroCluster configuration for use with disks and array LUNs, you must
consider various factors, such as the order of setting up access to storage, root aggregate location, and
the usage of FC initiator ports, switches, and FC-to-SAS bridges.
Consider the information in the following table when planning your configuration:

Consideration Guideline
Order of setting up access to the You can set up access to either disks or array LUNs first.
storage You must complete all setup for that type of storage and
verify that it is set up correctly before setting up the other
type of storage.
Location of the root aggregate If you are setting up a new MetroCluster deployment
with both disks and array LUNs, you must create the
root aggregate on native disks.
When doing this, ensure that at least one disk shelf
(with 24 disk drives) is set up at each of the sites.

If you are adding native disks to an existing

MetroCluster configuration that uses array LUNs, the
root aggregate can remain on an array LUN.
 Planning and installing a MetroCluster configuration with array LUNs | 191

Consideration Guideline
Using switches and FC-to-SAS FC-to-SAS bridges are required in four-node
bridges configurations and two-node fabric-attached
configurations to connect the Data ONTAP systems to
the disk shelves through the switches.
You must use the same switches to connect to the storage
arrays and the FC-to-SAS bridges.
Using FC initiator ports The initiator ports used to connect to an FC-to-SAS
bridge must be different from the ports used to connect to
the switches, which connect to the storage arrays.
A minimum of eight initiator ports is required to connect
a Data ONTAP system to both disks and array LUNs.

Related concepts
Example of switch zoning in a four-node MetroCluster configuration with array LUNs on page
173

Related tasks
Configuring the Cisco or Brocade FC switches manually on page 50
Installing FC-to-SAS bridges and SAS disk shelves on page 43

Related information
NetApp Hardware Universe

Examples of two-node stretch MetroCluster configurations with disks and

array LUNs
For setting up a stretch MetroCluster configuration with native disks and array LUNs, you must use
either FC-to-SAS bridges or SAS optical cables to connect the Data ONTAP systems to the disk
shelves. You must use FC switches for connecting array LUNs to the Data ONTAP systems.
A minimum of eight HBA ports is required for a Data ONTAP system to connect to both native disks
and array LUNs.
In the following examples representing two-node stretch MetroCluster configurations with disks and
array LUNs, HBA ports 0a through 0d are used for connection with array LUNs, while ports 1a
through 1d are used for connections with native disks.
The following illustration shows a two-node stretch MetroCluster configuration in which the native
disks are connected to the Data ONTAP systems through SAS optical cables:
192 | MetroCluster Installation and Configuration Guide

The following illustration shows a two-node stretch MetroCluster configuration in which the native
disks are connected to the Data ONTAP systems through FC-to-SAS bridges:

Note: If required, you can also use FC switches to connect native disks to the controllers in a
stretch MetroCluster configuration.

The following illustration shows a two-node stretch MetroCluster configuration with the array LUN
connections:
 Planning and installing a MetroCluster configuration with array LUNs | 193

Example of a two-node fabric-attached MetroCluster configuration with

disks and array LUNs
For setting up a two-node fabric-attached MetroCluster configuration with native disks and array
LUNs, you must use FC-to-SAS bridges to connect the Data ONTAP systems with the disk shelves
through the FC switches. You can connect array LUNs through the FC switches to the Data ONTAP
systems.
The following illustrations represent examples of a two-node fabric-attached MetroCluster
configuration with disks and array LUNs. They both represent the same MetroCluster configuration;
the representations for disks and array LUNs are separated only for simplification.
In the following illustration showing the connectivity between Data ONTAP systems and disks, the
HBA ports 1a through 1d are used for connectivity with disks through the FC-to-SAS bridges:

In the following illustration showing the connectivity between Data ONTAP systems and array
LUNs, the HBA ports 0a through 0d are used for connectivity with array LUNs because ports 1a
through 1d are used for connectivity with disks:
194 | MetroCluster Installation and Configuration Guide

Example of a four-node MetroCluster configuration with disks and array

LUNs
For setting up a four-node MetroCluster configuration with native disks and array LUNs, you must
use FC-to-SAS bridges to connect the Data ONTAP systems with the disk shelves through the FC
switches. You can connect array LUNs through the FC switches to the Data ONTAP systems.
A minimum of eight initiator ports is required for a Data ONTAP system to connect to both native
disks and array LUNs.
The following illustrations represent examples of a MetroCluster configuration with disks and array
LUNs. They both represent the same MetroCluster configuration; the representations for disks and
array LUNs are separated only for simplification.
In the following illustration that shows the connectivity between Data ONTAP systems and disks, the
HBA ports 1a through 1d are used for connectivity with disks through the FC-to-SAS bridges:

In the following illustration that shows the connectivity between Data ONTAP systems and array
LUNs, the HBA ports 0a through 0d are used for connectivity with array LUNs because ports 1a
through 1d are used for connectivity with disks:
Planning and installing a MetroCluster configuration with array LUNs | 195
196

Using the OnCommand management tools for

further configuration and monitoring
The OnCommand management tools can be used for GUI management of the clusters and
monitoring of the configuration.
Each node has OnCommand System Manager pre-installed. To load System Manager, enter the
cluster management LIF address as the URL in a web browser that has connectivity to the node.
You can also use OnCommand Unified Manager and OnCommand Performance Manager to monitor
the MetroCluster configuration.

Related information
NetApp Documentation: OnCommand Unified Manager Core Package (current releases)
NetApp Documentation: OnCommand System Manager (current releases)

Synchronizing the system time using NTP

Each cluster needs its own Network Time Protocol (NTP) server to synchronize the time between the
nodes and their clients. You can use the Edit DateTime dialog box in System Manager to configure
the NTP server.

Before you begin

You must have downloaded and installed System Manager. System Manager is available at
mysupport.netapp.com.

About this task

You cannot modify the time zone settings for a failed node or the partner node after a takeover
occurs.

Each cluster in the MetroCluster configuration should have its own separate NTP server or servers
used by the nodes, FC switches and FC-to-SAS bridges at that MetroCluster site.
If you are using the MetroCluster Tiebreaker software, it should also have its own separate NTP
server.

Steps

1. From the home page, double-click the appropriate storage system.

2. Expand the Cluster hierarchy in the left navigation pane.

3. In the navigation pane, click Configuration > System Tools > DateTime.

4. Click Edit.

5. Select the time zone.

6. Specify the IP addresses of the time servers, and then click Add.
You must add an NTP server to the list of time servers. The domain controller can be an
authoritative server.

7. Click OK.
 Using the OnCommand management tools for further configuration and monitoring | 197

8. Verify the changes you made to the date and time settings in the Date and Time window.
198

Requirements and limitations when using Data

ONTAP in a MetroCluster configuration
When using Data ONTAP in a MetroCluster configuration, you should be aware of certain
requirements and limitations when configuring Data ONTAP features.

Both sites should be licensed for the same site-licensed features.

All nodes should be licensed for the same node-locked features.

Infinite Volumes are not supported in a MetroCluster configuration.

Cluster peering from the MetroCluster sites to a third cluster

Because peering configuration is not replicated, if you peer one of the clusters in the MetroCluster
configuration to a third cluster outside of that configuration, you must also configure the peering on
the partner MetroCluster cluster. This ensures peering can be maintained in the event of a switchover.
The non-MetroCluster cluster must be running Data ONTAP 8.3 or peering will be lost in the event
of a switchover.

Volume creation on a root aggregate

The system does not allow the creation of new volumes on the root aggregate (an aggregate with an
HA policy of CFO) of a node in a MetroCluster configuration.
Because of this restriction, root aggregates cannot be added to an SVM using the vserver add-
aggregates command.

Networking and LIF creation guidelines for MetroCluster

configurations
You should be aware of how LIFs are created and replicated within the MetroCluster configuration.
You must also know about the requirement for consistency so you can make proper decisions when
configuring your network .

IPspace configuration
IPspace names must match between the two sites.
IPspace objects must be manually replicated to the partner cluster. Any SVMs created and assigned
to an IPspace before the IPspace is replicated will not be replicated to the partner cluster.

IPv6 configuration
If IPv6 is configured on one site, it must be configured on the other site.

LIF creation
You can confirm the successful creation of a LIF in a MetroCluster configuration by running the
metrocluster check lif show. If there are issues, you can use the metrocluster check
lif repair-placement command.
 Requirements and limitations when using Data ONTAP in a MetroCluster configuration | 199

Duplicate LIFs
You should not create duplicate LIFs (multiple LIFs with the same IP address) within the same
IPspace.

Intercluster LIFs
Intercluster LIFs are limited to the default IPspace that is owned by the admin SVM.

Replication of LIFs to the partner cluster

When you create a LIF on a cluster in a MetroCluster configuration, that LIF is replicated on the
partner cluster. The system must meet the following conditions to place the replicated LIF on the
partner cluster:

1. DR partner availability
The system attempts to place the replicated LIF on the DR partner of the node on which it was
created.

2. Connectivity
For IP or iSCSI LIFs, the system places them on a reachable subnet.

For FCP LIFs, the system attempts to place them on a reachable FC fabric.

3. Port attributes
The system attempts to place the LIF on a port with the desired VLAN, adapter type, and speed
attributes.

An EMS message is displayed if the LIF replication fails.

You can also check the failure by using the metrocluster check lif show command. Failures
can be corrected by running the metrocluster check lif repair-placement command for
any LIF that fails to find a correct port. You should resolve any LIF failures as soon as possible to
ensure LIF operation in the event of a MetroCluster switchover operation.
Note: Even if the source Storage Virtual Machine (SVM) is down, LIF placement may proceed
normally if there is a LIF belonging to a different SVM in a port with the same ipspace and
network in the destination.

Placement of replicated LIFs when the DR partner node is down

When a iSCSI or FCP LIF is created on a node whose DR partner has been taken over, the replicated
LIF is placed on the DR auxiliary partner node. After a subsequent giveback, the LIFs are not
automatically moved to the DR partner. This could lead to LIFs being concentrated on a single node
in the partner cluster. In the event of a MetroCluster switchover operation, subsequent attempts to
map LUNs belonging to the SVM will fail.
You should run the metrocluster check lif show after a takeover or giveback to ensure correct
LIF placement. If errors exist, you can run the metrocluster check lif repair-placement
command.

LIF placement errors

Starting in Data ONTAP 8.3.1, LIF placement errors displayed by the metrocluster check lif
show command will be retained after a switchover. If the network interface modify, network
interface rename, or network interface delete command is issued for a LIF with a
placement error, the error is removed and will not appear in the metrocluster check lif show
output.
200 | MetroCluster Installation and Configuration Guide

Related information
Clustered Data ONTAP 8.3 Network Management Guide

Volume or FlexClone command VLDB errors

If a volume or FlexClone volume command (such as volume create or volume delete) fails and the
error message indicates that the failure is due to a VLDB error, you should manually retry the job.
If the retry fails with an error that indicates a duplicate volume name, there is a stale entry in the
internal volume database. Please call customer support for assistance in removing the stale entry.
Removing the entry helps ensure that configuration inconsistencies do not develop between the two
MetroCluster clusters.

Output for storage disk show and storage shelf show

commands in a two-node MetroCluster configuration
In a two-node MetroCluster configuration, the is-local-attach field of the storage disk
show and storage shelf show commands shows all disks and storage shelves as local, regardless
of the node to which they are attached.

Modifying volumes to set NVFAIL in case of switchover

You can modify a volume so that, in event of a MetroCluster switchover, the NVFAIL flag is set on
the volume. The NVFAIL flag causes the volume to be fenced off from any modification. This is
required for volumes that need to be handled as if committed writes to the volume were lost after the
switchover.

Step

1. Enable MetroCluster to trigger NVFAIL on switchover by setting the vol -dr-force-nvfail

parameter to on:
vol modify -vserver vserver-name -volume volume-name -dr-force-nvfail on
.

Monitoring and protecting database validity by using

NVFAIL
The -nvfail parameter of the volume modify command enables Data ONTAP to detect
nonvolatile RAM (NVRAM) inconsistencies when the system is booting or after a switchover
operation. It also warns you and protects the system against data access and modification until the
volume can be manually recovered.
If Data ONTAP detects any problems, database or file system instances stop responding or shut
down. Data ONTAP then sends error messages to the console to alert you to check the state of the
database or file system. You can enable NVFAIL to warn database administrators of NVRAM
inconsistencies among clustered nodes that can compromise database validity.
After a system crash or switchover operation, NFS clients cannot access data from any of the nodes
until the NVFAIL state is cleared. CIFS clients are unaffected.
 Requirements and limitations when using Data ONTAP in a MetroCluster configuration | 201

How NVFAIL protects database files

The NVFAIL state is set in two cases, either when Data ONTAP detects NVRAM errors when
booting up or when a MetroCluster switchover operation occurs. If no errors are detected at startup,
the file service is started normally. However, if NVRAM errors are detected or the force-fail
option was set and then there was a switchover, Data ONTAP stops database instances from
responding.
When you enable the NVFAIL option, one of the following processes takes place during bootup:.

If... Then...
Data ONTAP detects no NVRAM errors File service starts normally.
Data ONTAP detects NVRAM errors Data ONTAP returns a stale file handle
(ESTALE) error to NFS clients trying to
access the database, causing the application
to stop responding, crash, or shut down.
Data ONTAP then sends an error message to
the system console and log file.

When the application restarts, files are

available to CIFS clients, even if you have
not verified that they are valid.
For NFS clients, files remain inaccessible
until you reset the in-nvfailed-state
option on the affected volume.

Data ONTAP detects NVRAM errors on a LUNs in that volume are brought offline. Then
volume that contains LUNs the in-nvfailed-state option on the
volume must be cleared, and the NVFAIL
attribute on the LUNs must be cleared by
bringing each LUN in the affected volume
online. You can perform the steps to check the
integrity of the LUNs and recover the LUN
from Snapshot copy or backup as necessary.
After all the LUNs in the volume are recovered,
the in-nvfailed-state option on the
affected volume is cleared.

Commands for monitoring data loss events

If you enable the NVFAIL option, you receive notification when a system crash caused by NVRAM
inconsistencies or a MetroCluster switchover occurs.
By default, the NVFAIL parameter is not enabled.

If you want to... Use this command...

Create a new volume with NVFAIL enabled volume create -nvfail on

Enable NVFAIL on an existing volume volume modify

Note: You set the -nvfail option to on to

enable NVFAIL on the created volume.
202 | MetroCluster Installation and Configuration Guide

If you want to... Use this command...

Display whether NVFAIL is currently enabled volume show
for a specified volume
Note: You set the -fields parameter to
nvfail to display the NVFAIL attribute for a
specified volume.

See the man page for each command for more information.

Accessing volumes in NVFAIL state after a switchover

After a switchover, you must clear the NVFAIL state by resetting the -in-nvfailed-state
parameter of the volume modify command to remove the restriction of clients to access data.

Before you begin

The database or file system must not be running or trying to access the affected volume.

About this task

Setting -in-nvfailed-state parameter requires advanced-level privilege.

Step

1. Recover the volume by using the volume modify command with the -in-nvfailed-state
parameter set to false.

After you finish

For instructions about examining database file validity, see the documentation for your specific
database software.
If your database uses LUNs, review the steps to make the LUNs accessible to the host after an
NVRAM failure.

Recovering LUNs in NVFAIL states after switchover

After a switchover, the host no longer has access to data on the LUNs that are in NVFAIL states. You
must perform a number of actions before the database has access to the LUNs.

Before you begin

The database must not be running.

Steps

1. Clear the NVFAIL state on the affect volume that hosts the LUNs by resetting the -in-
nvfailed-state parameter of the volume modify command.

2. Bring the affected LUNs online.

3. Examine the LUNs for any data inconsistencies and resolve them.
This might involve host-based recovery or recovery done on the storage controller using
SnapRestore.

4. Bring the database application online after recovering the LUNs.

 203

Glossary of MetroCluster terms

aggregate
A grouping of physical storage resources (disks or array LUNs) that provides storage to
volumes associated with the aggregate. Aggregates provide the ability to control the RAID
configuration for all associated volumes.
data SVM
Formerly known as data Vserver. In clustered Data ONTAP, a Storage Virtual Machine
(SVM) that facilitates data access from the cluster; the hardware and storage resources of
the cluster are dynamically shared by data SVMs within a cluster.
admin SVM
Formerly known as admin Vserver. In clustered Data ONTAP, a Storage Virtual Machine
(SVM) that has overall administrative access to all objects in the cluster, including all
objects owned by other SVMs, but does not provide data access to clients or hosts.
inter-switch link (ISL)
A connection between two switches using the E-port.
destination
The storage to which source data is backed up, mirrored, or migrated.
disaster recovery (DR) group
The four nodes in a MetroCluster configuration that synchronously replicate each others'
configuration and data.
disaster recovery (DR) partner
A node's partner at the remote MetroCluster site. The node mirrors its DR partner's
NVRAM or NVMEM partition.
disaster recovery auxiliary (DR auxiliary) partner
The HA partner of a node's DR partner. The DR auxiliary partner mirrors a node's
NVRAM or NVMEM partition in the event of an HA takeover after a MetroCluster
switchover operation.
HA pair
In Data ONTAP 8.x, a pair of nodes whose controllers are configured to serve data for
each other if one of the two nodes stops functioning.
Depending on the system model, both controllers can be in a single chassis, or one
controller can be in one chassis and the other controller can be in a separate chassis.

In the Data ONTAP 7.3 and 7.2 release families, this functionality is referred to as an
active/active configuration.
HA partner
A node's partner within the local HA pair. The node mirrors its HA partner's NVRAM or
NVMEM cache.
high availability (HA)
In Data ONTAP 8.x, the recovery capability provided by a pair of nodes (storage systems),
called an HA pair, that are configured to serve data for each other if one of the two nodes
stops functioning. In the Data ONTAP 7.3 and 7.2 release families, this functionality is
referred to as an active/active configuration.
healing
204 | MetroCluster Installation and Configuration Guide

The two required MetroCluster operations that prepare the storage located at the DR site
for switchback. The first heal operation resynchronizes the mirrored plexes. The second
heal operation returns ownership of root aggregates to the DR nodes.
LIF (logical interface)
A logical network interface, representing a network access point to a node. LIFs currently
correspond to IP addresses, but could be implemented by any interconnect. A LIF is
generally bound to a physical network port; that is, an Ethernet port. LIFs can fail over to
other physical ports (potentially on other nodes) based on policies interpreted by the LIF
manager.
NVRAM
nonvolatile random-access memory.
NVRAM cache
Nonvolatile RAM in a storage system, used for logging incoming write data and NFS
requests. Improves system performance and prevents loss of data in case of a storage
system or power failure.
NVRAM mirror
A synchronously updated copy of the contents of the storage system NVRAM (nonvolatile
random access memory) contents kept on the partner storage system.
node
In Data ONTAP, one of the systems in a cluster or an HA pair.
To distinguish between the two nodes in an HA pair, one node is sometimes called the
local node and the other node is sometimes called the partner node or remote node.
In Protection Manager and Provisioning Manager, the set of storage containers
(storage systems, aggregates, volumes, or qtrees) that are assigned to a dataset and
designated either primary data (primary node), secondary data (secondary node), or
tertiary data (tertiary node).
A dataset node refers to any of the nodes configured for a dataset.
A backup node refers to either a secondary or tertiary node that is the destination of a
backup or mirror operation.
A disaster recovery node refers to the dataset node that is the destination of a failover
operation.

remote storage
The storage that is accessible to the local node, but is at the location of the remote node.
root volume
A special volume on each Data ONTAP system. The root volume contains system files
and configuration information, and can also contain data. It is required for the system to
be able to boot and to function properly. Core dump files, which are important for
troubleshooting, are written to the root volume if there is enough space.
switchback
The MetroCluster operation that restores service back to one of the MetroCluster sites.
switchover
The MetroCluster operation that transfers service from one of the MetroCluster sites.

A negotiated switchover is planned in advance and cleanly shuts down components of

the target MetroCluster site.

A forced switchover immediately transfers service; the shut down of the target site
might not be clean.
 205

Copyright information
Copyright 19942015 NetApp, Inc. All rights reserved. Printed in the U.S.
No part of this document covered by copyright may be reproduced in any form or by any means
graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an
electronic retrieval systemwithout prior written permission of the copyright owner.
Software derived from copyrighted NetApp material is subject to the following license and
disclaimer:
THIS SOFTWARE IS PROVIDED BY NETAPP "AS IS" AND WITHOUT ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE,
WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
NetApp reserves the right to change any products described herein at any time, and without notice.
NetApp assumes no responsibility or liability arising from the use of products described herein,
except as expressly agreed to in writing by NetApp. The use or purchase of this product does not
convey a license under any patent rights, trademark rights, or any other intellectual property rights of
NetApp.
The product described in this manual may be protected by one or more U.S. patents, foreign patents,
or pending applications.
RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to
restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer
Software clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).
206

Trademark information
NetApp, the NetApp logo, Go Further, Faster, AltaVault, ASUP, AutoSupport, Campaign Express,
Cloud ONTAP, Clustered Data ONTAP, Customer Fitness, Data ONTAP, DataMotion, Fitness, Flash
Accel, Flash Cache, Flash Pool, FlashRay, FlexArray, FlexCache, FlexClone, FlexPod, FlexScale,
FlexShare, FlexVol, FPolicy, GetSuccessful, LockVault, Manage ONTAP, Mars, MetroCluster,
MultiStore, NetApp Insight, OnCommand, ONTAP, ONTAPI, RAID DP, RAID-TEC, SANtricity,
SecureShare, Simplicity, Simulate ONTAP, Snap Creator, SnapCenter, SnapCopy, SnapDrive,
SnapIntegrator, SnapLock, SnapManager, SnapMirror, SnapMover, SnapProtect, SnapRestore,
Snapshot, SnapValidator, SnapVault, StorageGRID, Tech OnTap, Unbound Cloud, and WAFL and
other names are trademarks or registered trademarks of NetApp, Inc., in the United States, and/or
other countries. All other brands or products are trademarks or registered trademarks of their
respective holders and should be treated as such. A current list of NetApp trademarks is available on
the web at http://www.netapp.com/us/legal/netapptmlist.aspx.
 207

How to send comments about documentation and

receive update notifications
You can help us to improve the quality of our documentation by sending us your feedback. You can
receive automatic notification when production-level (GA/FCS) documentation is initially released or
important changes are made to existing production-level documents.
If you have suggestions for improving this document, send us your comments by email to
[email protected]. To help us direct your comments to the correct division, include in the
subject line the product name, version, and operating system.
If you want to be notified automatically when production-level documentation is released or
important changes are made to existing production-level documents, follow Twitter account
@NetAppDoc.
You can also contact us in the following ways:

NetApp, Inc., 495 East Java Drive, Sunnyvale, CA 94089 U.S.

Telephone: +1 (408) 822-6000

Fax: +1 (408) 822-4501

Support telephone: +1 (888) 463-8277

208 | MetroCluster Installation and Configuration Guide

Index
supported MetroCluster configurations 11
7-Mode fabric MetroClusters
sharing FC switch fabrics 20
7-Mode MetroCluster configurations B
differences from clustered MetroCluster
configurations 10 bridges
installing FC-to-SAS 43, 100
Brocade
A license requirements 52, 108
Brocade 6510s
addresses sharing during transition 107
configuring NTP server, to synchronize system time Brocade FC switch configuration
196 configuring ISL ports 61
gathering required network information 117, 119 configuring zoning 62
aggregates setting to default values 52
mirrored data, creating on each node of a Brocade FC switch configurations
MetroCluster configuration 145, 184 configuring ISL ports 57
aggregates, root Brocade FC switches
mirroring 183 configuring 50
architecture configuring basic settings 55
of two-node MetroCluster configurations 87, 95 introduction to manually configuring 50
parts of fabric MetroCluster 23 requirements for configuring 50
array LUNs Brocade switches
cabling ports in stretch MetroCluster configurations enabling ISL encryption 67
with 160 setting ISL encryption 65
configuration requirements for MetroCluster setting payload for 65
configurations 155 setting the authentication policy 66
in a MetroCluster configuration 14
installing and cabling MetroCluster components for
156 C
installing license for using 180
introduction to implementing a MetroCluster cabling
configuration with disks and 190 data ports 43, 92, 99, 167
planning for a MetroCluster configuration with 154 HA interconnects 42, 166
supported MetroCluster configurations with 154 management ports 43, 92, 99, 167
switch ports required for a four-node MetroCluster MetroCluster components 42, 166
configuration with 158 MetroCluster components, array LUNs 156
switch ports required for a MetroCluster stretch MetroCluster systems 89
configuration with 158 cabling)
switch ports required for a two-node MetroCluster Inter-Switch Links 37, 164
configuration with 158 cascade configurations
array LUNs and disks networking requirements for cluster peering 12
considerations when implementing MetroCluster checking
configuration with 190 MetroCluster configuration operations 149, 188
example four-node MetroCluster configuration 194 checklists
example two-node fabric-attached MetroCluster software setup, for factory-configured MetroCluster
configuration 193 17
example two-node stretch MetroCluster checklists, hardware setup
configurations 191 for factory configured clusters 15
arrays, storage Cisco 9148 switches
preparing for use with Data ONTAP systems 157 manually enabling ports 72
assigning disk ownership in AFF systems Cisco FC switch configuration
MetroCluster configuration 126 calculating buffer-to-buffer credits 80
assigning disk ownership in non-AFF systems community string 70
MetroCluster configuration 125 configuring ISL ports 74, 80
ATTO FibreBridge configuring port channels 80
See FC-to-SAS bridges configuring VSANs 76
authentication policy setting basic settings 70
setting 66 setting to default values 70
automatic switchover Cisco FC switch configurations
 Index | 209

configuring zoning 83 checking for common configuration errors 151

Cisco FC switches downloading and running 151
configuration requirements 68 configuration backup files
configuring with configuration files 50 setting remote destinations for preservation 153
introduction to manually configuring 50 configuration files
ISL requirements 68 configuring Brocade FC switches 49
storage connection requirements 68 configuring Cisco FC switches with 50
Cisco switch configuration configuration networking
saving 86 how LIFs are created 198
Cisco switches how LIFs are replicated 198
manually enabling ports in MDS 9148 72 IPv6 configuration 198
port licensing 71 configurations
cluster configurations cascade, for cluster peering 12
hardware and software 122 fan-out, for cluster peering 12
regular and MetroCluster 122 implementing MetroCluster configurations 146, 185
cluster interconnects configurations, MetroCluster
cabling MetroCluster configurations 41, 165 array LUN requirements 155
cluster peer relationships introduction to implementing with disks and array
requirements for 12 LUNs 190
cluster peering configurations, site
illustration 28 worksheets for 117, 119
introduction to MetroCluster configuration 135, 181 configuring
MetroCluster 28 NTP server, to synchronize system time 196
to an outside cluster 198 considerations
cluster peering connections MetroCluster configuration with disks and array
cabling in MetroCluster configurations 41, 92, 99, LUNs 190
166 controller ports
cluster peers checking connectivity with the partner site 41, 92,
creating relationships between 143, 182 99, 166
cluster setup controllers
introduction to the process for 179 racking 90
Cluster Setup wizard racking in a MetroCluster 98
using to set up a cluster in a two-node MetroCluster racking in the MetroCluster 33, 108
configuration 133 verifying and setting HA state 129, 176
clustered MetroCluster configurations creation, LIFs
differences between the types 11 in a MetroCluster configuration 198
differences from 7-Mode MetroCluster
configurations 10
clusters
D
example names in MetroCluster configuration 88, 94 data aggregates
example names in the MetroCluster configuration 29 mirrored, creating on each node of a MetroCluster
introduction to manually peering 137 configuration 145, 184
naming requirements for cluster peering 12 data field size
setting up in a two-node MetroCluster configuration setting 65
133 data ports
clusters, factory configured cabling 43, 92, 99, 167
hardware setup checklist 15 configuring intercluster LIFs to share 141
CNA adapter settings considerations when sharing intercluster and 13
configuring when reusing a controller module 122 database files
commands how NVFAIL protects 201
for starting data protection 146, 185 databases
volume 201 accessing after a switchover 202
comments introduction to using NVFAIL to monitor and
how to send feedback about documentation 207 protect validity of 200
community string dedicated ports
FC switches 70 considerations when using for intercluster replication
Health Monitors 70 13
components defaults
preconfigured when new 15 restoring when reusing a controller module 122
racking 90 destinations
racking in a MetroCluster 98 specifying the URL for configuration backup 153
racking in the MetroCluster 33, 108 disabling
Config Advisor virtual fabric in a Brocade switch 65
210 | MetroCluster Installation and Configuration Guide

disaster recovery group 28 configuring ports

disk assignment Cisco 73
verifying in a MetroCluster configuration 128 configuring VSANs
disk assignments Cisco 76
verifying in a four-node MetroCluster configuration setting basic settings
123 Cisco 70
disk ownership community string 70
assigning array LUNs 180 setting to default values
disk ownership in AFF systems Brocade 52
assigning in a MetroCluster configuration 126 Cisco 70
disk ownership in non-AFF systems switch names
assigning in a MetroCluster configuration 125 setting for Brocade switches 52
disk shelves FC switch configurations
racking 90 configuring basic settings for Brocade 55
racking in a MetroCluster 98 downloading configuration files, Brocade 49
racking in the MetroCluster 33, 108 recommended port assignments 38, 40
diskless systems) worksheets 31
configuring Data ONTAP for 177 FC switch fabric
installing Data ONTAP for 177 reenabling 114
setting up Data ONTAP on 177 FC switch fabrics
disks redundant configuration in the MetroCluster
introduction to implementing a MetroCluster architecture 28
configuration with array LUNs and 190 sharing during 7-Mode transition 107
disks and array LUNs FC switches
considerations when implementing MetroCluster cabling HBA adapters 34
configuration with 190 cabling storage arrays to in a MetroCluster
documentation configuration 167
how to receive automatic notification of changes to configuration requirements for MetroCluster
207 configurations with array LUNs 155
how to send feedback about 207 configuring Cisco, with configuration files 50
where to find MetroCluster documentation 8 configuring for health monitoring 148, 187
configuring zoning on Cisco FC switches 83
example names in MetroCluster configuration 88, 94
E example names in the MetroCluster configuration 29
E-ports introduction to configuring 49
configuring 57 introduction to manually configuring Cisco and
events Brocade 50
monitoring data loss 201 introduction to recommended port assignments 38
example names racking 90, 98
MetroCluster component 88, 94 racking in a MetroCluster configuration with array
MetroCluster components 29 LUNs 157
racking in the MetroCluster 33, 108
FC switches, Brocade
F configuring 50
requirements for configuring 50
fabric-attached MetroCluster configurations
FC switches, Cisco
array LUN requirements 155
configuration requirements 68
example two-node configuration with array LUNs
ISL requirements 68
and disks 193
storage connection requirements 68
fabrics, Brocade FC switch
FC-to-SAS bridge configurations
configuring 50
worksheets 31, 97
requirements for configuring 50
FC-to-SAS bridge ports
factory configured clusters
cabling during 7-Mode transition 109
hardware setup checklist 15
FC-to-SAS bridges
failover and giveback
example names in MetroCluster configuration 88, 94
verifying in a MetroCluster configuration 151
example names in the MetroCluster configuration 29
fan-out configurations
in the MetroCluster architecture 27
networking requirements for cluster peering 12
installing 43, 100, 102
FC switch
meeting preinstallation requirements 44, 101
verifying 114
FC-VI ports
FC switch configuration
cabling 91, 99
configuring ISL ports
cabling during 7-Mode transition 109
Cisco 74
cabling in four-node configurations 162
 Index | 211

cabling in two-node configurations 161 FC switch fabrics in the MetroCluster architecture

feedback 28
how to send comments about documentation 207 FC-to-SAS bridges in the MetroCluster architecture
FibreBridge 27
See FC-to-SAS bridges Infinite Volumes
FibreBridge bridges in a MetroCluster configuration 198
installing 102 information
files, database how to send feedback about improving
how NVFAIL protects 201 documentation 207
firewalls initial configuration
requirements for cluster peering 12 performing with the System Setup tool 130
FlexClone command errors installation
in MetroCluster configurations 200 for systems sharing an FC switch fabric 20
four-node MetroCluster configuration for systems with array LUNs 20
switch ports required for a 158 for systems with native disks 20
four-node MetroCluster configurations installing FC-to-SAS bridges 102
cabling array LUNs and switches in 169 installing new FibreBridge bridges 102
cabling FC-VI ports in 162 preparations for installing FC-to-SAS bridges 44,
cabling HBA ports in 162 101
full-mesh connectivity installing
description 12 MetroCluster components, array LUNs 156
Inter-Switch Link ports
cabling 37, 164
H Inter-Switch Links
HA interconnects cabling 37, 164
cabling 42, 166 intercluster LIFs
HA pair operations configuring to share data ports 141
verifying in a MetroCluster configuration 151 configuring to use dedicated intercluster ports 137
HA pairs considerations when sharing with data ports 13
illustration of local MetroCluster 27 intercluster networks
MetroCluster configurations configuring intercluster LIFs for 137, 141
illustration of local HA pairs 27 considerations when sharing data and intercluster
HA state ports 13
verifying and setting controllers 129, 176 intercluster ports
hardware components configuring intercluster LIFs to use dedicated 137
racking 90 considerations when using dedicated 13
racking in a MetroCluster 98 IOD settings
racking in a MetroCluster configuration with array deleting TI zoning and configuring 112
LUNs 157 IP addresses
racking in the MetroCluster 33, 108 gathering required network information 117, 119
hardware setup requirements for cluster peering 12
checklist for factory configured clusters 15 IPspaces
HBA adapter ports requirements for cluster peering 12
cabling 34 ISL encryption
HBA ports enabling in a MetroCluster configuration 67
cabling during 7-Mode transition 109 enabling in switches 67
cabling in four-node configurations 162 setting on Brocade switches 65
cabling in two-node configurations 161 ISL port group
HBA settings sharing a switch fabric 113
configuring when reusing a controller module 122 ISL ports
healing configuring 57
verifying in a MetroCluster configuration 153 configuring on Brocade switches 61
health monitoring configuring on Cisco switches 80
configuring FC switches for 148, 187 ISLs
Health Monitors See Inter-Switch Links
community string 70
host names L
gathering required network information 117, 119
license
installing it for using array LUNs, MetroCluster
I configuration 180
illustration licensing
in a MetroCluster configuration 198
212 | MetroCluster Installation and Configuration Guide

LIF creation racking storage controllers in a 157

in a MetroCluster configuration 198 requirements for zoning in 171
LIF replication setting up Data ONTAP in a 175
in a MetroCluster configuration 198 switch zoning in a 171
LIFs MetroCluster configuration with array LUNS
configuring to use dedicated intercluster ports 137 switch ports required for a 158
preconfigured in a MetroCluster configuration 135 MetroCluster configuration with array LUNS, four-node
LIFs, intercluster switch ports required for a 158
configuring to share data ports 141 MetroCluster configuration with array LUNS, two-node
local HA switch ports required for a 158
supported MetroCluster configurations 11 MetroCluster configuration with disks and array LUNs
local HA pairs planning and installing a 154
illustration of MetroCluster 27 MetroCluster configurations
verifying operation of, in a MetroCluster architecture of two-node configuration 87
configuration 151 array LUN requirements 155
LUNs cabling array LUNs and switches 167
recovering after NVRAM failures 202 cabling FC-VI adapters 99
LUNs (array) cabling HBA adapters 34, 99
assigning ownership of 180 cabling Inter-Switch Links 37, 164
LUNs, array cabling MetroCluster components
configuration requirements for MetroCluster cabling FC-VI adapters
configurations 155 cabling HBA adapters 91
introduction to implementing a MetroCluster creating mirrored data aggregates on each node of
configuration with disks and 190 145, 184
LUNs(array) enabling ISL encryption on Brocade switches 67
configuring Data ONTAP for 177 example four-node configuration with array LUNs
installing Data ONTAP for 177 and disks 194
illustration of 23
implementing 146, 185
M installing license for array LUNs in 180
management ports introduction to gathering required information and
cabling 43, 92, 99, 167 reviewing the workflow 96
manually peering clusters introduction to implementing with disks and array
introduction to 137 LUNs 190
MetroCluster parts of 23
sharing existing switch fabric 107 setting ISL encryption on Brocade switch 65
MetroCluster architecture setting the authentication policy for Brocade
cluster peering 28 switches 66
FC switch fabrics 28 supported with array LUNs 154
FC-to-SAS bridges 27 verifying correct operation of 149, 188
illustration MetroCluster configurations, two-node
cluster peering in the MetroCluster architecture illustration of 95
28 parts of 95
MetroCluster components metrocluster configure command
cabling cluster interconnects 41, 165 creating MetroCluster relationships 146, 185
cabling the HA interconnect 42, 166 MetroCluster monitoring
installing and cabling 156 using Tiebreaker software 153
racking disk shelves 33, 90, 98, 108 MetroCluster systems
racking FC switches 33, 90, 98, 108 cabling with SAS optical cables and SAS shelves 89
racking storage controllers 33, 90, 98, 108 mirroring
MetroCluster configuration root aggregates 183
bridge-attached stretch, workflow for cabling two
nodes 94 N
fabric-attached, workflow for cabling four or two
nodes 22 native disk shelves
SAS-attached stretch, workflow for cabling two in a MetroCluster configuration 14
nodes 87 network
switch ports required for a 158 full-mesh connectivity described 12
MetroCluster configuration with array LUNs requirements for cluster peering 12
cabling FC-VI ports in 159 network information
cabling HBA ports in 159 gathering required 117, 119
planning and installing a 154 node configuration after MetroCluster setup
racking FC switches in a 157
 Index | 213

additional configuration with OnCommand System Cisco switches 71

Manager 196 port numbers
nodes Brocade FC switches 38, 40
creating mirrored data aggregates on each Cisco FC switch 38, 40
MetroCluster 145, 184 port usage
using the Cluster Setup wizard to set up, in a two- in a MetroCluster configuration 135
node MetroCluster configuration 133 ports
NTP servers configuring intercluster LIFs to use dedicated
configuring to synchronize system time 196 intercluster 137
NVFAIL considerations when sharing data and intercluster
description of 200 roles on 13
how it protects database files 201 considerations when using dedicated intercluster 13
modifying volumes to set NVFAIL in case of manually enabling in Cisco MDS 9148 switches 72
switchover 200 requirements for cluster peering 12
NVRAM failures ports, data
recovering LUNs after 202 configuring intercluster LIFs to share 141
preconfiguration
MetroCluster components 15
O preparing storage arrays
OnCommand Performance Manager for use with Data ONTAP systems 157
monitoring with 196
OnCommand System Manager Q
node configuration with 196
OnCommand Unified Manager QoS policies
monitoring with 196 VSANs on Cisco switches 76
ownership
assigning array LUNs 180
R
P relationships
creating cluster peer 143, 182
partner cluster creating MetroCluster relationships 146, 185
cabling cluster peering connections 41, 92, 99, 166 replication, LIFs
passwords in a MetroCluster configuration 198
preconfigured in a MetroCluster configuration 15 required hardware
payload MetroCluster 29, 88, 94
setting 65 requirements
peer relationships cluster naming when peering 12
creating cluster 143, 182 firewall for cluster peering 12
requirements for clusters 12 gathering network information 117, 119
peering clusters IP addresses for cluster peering 12
introduction to manually 137 IPspaces for cluster peering 12
MetroCluster configuration, introduction to 135, 181 network for cluster peering 12
planning ports for cluster peering 12
gathering required network information 117, 119 subnets for cluster peering 12
pool0 disks root aggregates
assigning in a MetroCluster configuration with AFF mirroring 145, 183
systems 126 volume creation on 198
assigning in a MetroCluster configuration with non-
AFF systems 125
pool1 disks
S
assigning in a MetroCluster configuration with AFF SAS disk shelves
systems 126 installing 43, 100
assigning in a MetroCluster configuration with non- SAS optical cables
AFF systems 125 cabling in stretch MetroCluster systems with SAS
port assignments shelves 89
recommended FC switch, introduction to 38 SAS shelves
port channels cabling in MetroCluster systems with SAS optical
configuring on Cisco switches 80 cables 89
port configuration servers
on FC switch configuring NTP, to synchronize system time 196
Cisco 73 setting
port licensing
214 | MetroCluster Installation and Configuration Guide

authentication policy for a Brocade switch in a MetroCluster configuration with array LUNs 171
MetroCluster configuration 66 requirements in MetroCluster configurations for 171
setup switchback
software, checklist for factory configured verifying in a MetroCluster configuration 153
MetroCluster 17 switches
setup, hardware licensing requirements
checklist for factory configured clusters 15 Brocade 52, 108
site configurations Cisco 69
worksheets for 117, 119 switches, Brocade FC
software configuring 50
settings already enabled 15 requirements for configuring 50
setup checklist for factory configured clusters 17 switches, Cisco 9148
software, configuring manually enabling ports 72
workflows for MetroCluster in Data ONTAP 116 switches, Cisco FC
storage arrays configuration requirements 68
cabling to FC switches in a MetroCluster ISL requirements 68
configuration 167 storage connection requirements 68
configuration requirements for MetroCluster switches, FC
configurations 155 configuration requirements for MetroCluster
preparing for use with Data ONTAP systems 157 configurations with array LUNs 155
storage controllers introduction to configuring 49
example names in MetroCluster configuration 88, 94 introduction to manually configuring Cisco and
example names in the MetroCluster configuration 29 Brocade 50
racking in a Metrocluster configuration 157 switchover
storage disk show command accessing the database after 202
output in a two-node MetroCluster configuration 200 verifying in a MetroCluster configuration 153
storage shelf show command synchronizing system time
output in a two-node MetroCluster configuration 200 using NTP 196
stretch MetroCluster configurations SyncMirror
cabling FC-VI ports 160 configuration requirements for MetroCluster
cabling HBA ports 160 configurations with array LUNs 155
example configurations with array LUNs and disks System Setup tool
191 performing basic software configuration 130
subnets performing cluster configuration 130
requirements for cluster peering 12 system time
suggestions synchronizing using NTP 196
how to send feedback about documentation 207
SVMs
preconfigured in a MetroCluster configuration 135
T
switch configuration, Cisco third-party storage
saving 86 requirements for MetroCluster configurations with
switch configurations, FC array LUNs 155
configuring basic settings for Brocade 55 TI zoning
switch fabrics deleting and configuring IOD settings 112
configuring switch ports for 57 Tiebreaker software
disabling before modifying configuration 110 installing 153
enabling sharing for a port group 113 using to identify failures 153
Fibre Channel 28 time
reenabling 110 synchronizing system, using NTP 196
switch parameters transition
setting 65 7-Mode to clustered Data ONTAP 14
switch ports sharing FC switch fabric 14
required for a four-node MetroCluster configuration twitter
with array LUNs 158 how to receive automatic notification of
required for a MetroCluster configuration with array documentation changes 207
LUNs 158 two-node configurations
required for a two-node MetroCluster configuration parts of 95
with array LUNs 158 two-node MetroCluster configuration
switch zoning switch ports required for a 158
example of four-node MetroCluster configuration two-node MetroCluster configurations
with array LUNs 173 cabling array LUNs and switches in 168
example of two-node MetroCluster configuration cabling FC-VI ports in 161
with array LUNs 172 cabling HBA ports in 161
 Index | 215

U Cisco 76
Vservers
usernames See SVMs
preconfigured in a MetroCluster configuration 15
utilities
checking for common configuration errors with W
Config Advisor 151 workflows
downloading and running Config Advisor 151 cabling a four-node or two-node fabric-attached
MetroCluster configuration 22
V cabling a two-node bridge-attached stretch
MetroCluster configuration 94
verification cabling a two-node SAS-attached stretch
booting to Maintenance mode 123, 128 MetroCluster configuration 87
performing before booting to Data ONTAP 123, 128 MetroCluster software configurations in Data
verifying disk assignment 123, 128 ONTAP 116
verifying worksheets
MetroCluster configuration operations 149, 188 for FC switch configurations 31
Virtual fabric for FC-to-SAS bridge configurations 31, 97
disabling 65 for site configurations 117, 119
VLDB errors
in MetroCluster configurations 200
volume command errors Z
in MetroCluster configurations 200 zoning
volume creation sharing a switch fabric 113
in a MetroCluster configuration 198 zoning configuration
volumes configuring on a Brocade FC switch 62
commands 201 zoning configurations
recovering after a switchover 202 configuring on a Cisco FC switch 83
VSANs