WHITE PAPER

ISILON NETWORK DESIGN CONSIDERATIONS

ABSTRACT
This white paper explores design considerations of Isilon’s external network to ensure
maximum performance and an optimal user experience.

Version 4.3

November 2018

Isilon Network Design Considerations

© 2018 Dell Inc.
 TABLE OF CONTENTS

EXECUTIVE SUMMARY ...........................................................................................................5

NOTICE TO READERS .............................................................................................................5
VERSION HISTORY ..................................................................................................................5
NETWORK ARCHITECTURE DESIGN.....................................................................................6
General Network Architecture Considerations .................................................................................. 6
Triangle Looped Topology ................................................................................................................. 7
Link Aggregation ............................................................................................................................... 8
Multi-Chassis Link Aggregation .................................................................................................................. 8

LATENCY, BANDWIDTH & THROUGHPUT ............................................................................9

Latency .............................................................................................................................................. 9
Bandwidth & Throughput ................................................................................................................... 9
Bandwidth Delay Product ......................................................................................................................... 10
Isilon Network Stack Tuning ..................................................................................................................... 11

ETHERNET FLOW CONTROL............................................................................................... 12

SYNCIQ CONSIDERATIONS ................................................................................................. 13
SyncIQ Disaster Recovery with SmartConnect ............................................................................... 13
Replication Traffic Over Dedicated WAN Links ............................................................................... 13

ISILON ONEFS PORTS .......................................................................................................... 13

SMARTCONNECT CONSIDERATIONS ................................................................................ 14
SmartConnect Network Hierarchy ................................................................................................... 14
Load Balancing................................................................................................................................ 15
Static Versus Dynamic IP Address Allocation ................................................................................. 15
Dynamic Failover............................................................................................................................. 15
Dynamic Failover Examples ............................................................................................................ 16
Dynamic Zone with 1 IP Address Per Node ............................................................................................. 16
Dynamic Zone with 3 IP Address Per Node ............................................................................................. 17
Protocols and SmartConnect Allocation Methods ........................................................................... 18
SMB ......................................................................................................................................................... 18
NFS ......................................................................................................................................................... 18
HDFS ....................................................................................................................................................... 18
Suggested Zones By Protocol .................................................................................................................. 18
IP Address Quantification ................................................................................................................ 19

2 | Isilon Network Design Considerations

© 2018 Dell Inc.
 Smartconnect Node Suspension ..................................................................................................... 20
SmartConnect and Reverse DNS .................................................................................................... 20
DNS Delegation Best Practices ....................................................................................................... 20
SmartConnect in Isolated Network Environments ........................................................................... 21
SmartConnect DNS, Subnet, and Pool Design ............................................................................... 22
SmartConnect Zone Naming .................................................................................................................... 22
SmartConnect with Multiple Node Pools or Types.................................................................................... 22
Where the SmartConnect Service IP Runs ..................................................................................... 23
Other SmartConnect Considerations............................................................................................... 24

ETHERNET, MTU, & IP OVERHEAD ..................................................................................... 25

Ethernet Packet ............................................................................................................................... 25
Ethernet Payload ..................................................................................................................................... 26
Jumbo Frames ................................................................................................................................ 26
IP Packet Overhead ........................................................................................................................ 27
Example 1: Standard 1500 Byte Payload – IPv4 / TCP ............................................................................ 27
Example 2: Jumbo 9000 Byte Payload – IPv4 / TCP................................................................................ 27
Example 3: Standard 1500 Byte Payload – IPv4 / TCP / Linux TimeStamp ............................................. 28
Example 4: Jumbo 9000 Byte Payload – IPv4 / TCP / Linux Timestamp .................................................. 28
Data Payload to Ethernet Frame Effeciency............................................................................................. 28
ICMP and MTU With Isilon .............................................................................................................. 29
Isilon OneFS MTU Commands........................................................................................................ 29
Confirming Transmitted MTU .......................................................................................................... 29

ACCESS ZONES BEST PRACTICES.................................................................................... 30

System Zone ................................................................................................................................... 30
Root Based Path ............................................................................................................................. 30

SOURCE-BASED ROUTING CONSIDERATIONS ................................................................ 31

Source-Based Routing & DNS ........................................................................................................ 32

IPV6 ......................................................................................................................................... 33
Why IPv6? ....................................................................................................................................... 33
Security.................................................................................................................................................... 33
Efficiency ................................................................................................................................................. 33
Multicast .................................................................................................................................................. 33
Quality of Service ..................................................................................................................................... 33
IPv6 Addressing .............................................................................................................................. 34

3 | Isilon Network Design Considerations

© 2018 Dell Inc.
 IPv6 Header .................................................................................................................................... 35
IPv6 to IPv4 Translation .................................................................................................................. 35
Configuring Isilon OneFS for IPv6 ................................................................................................... 35

NETWORK TROUBLESHOOTING ........................................................................................ 36

Netstat ............................................................................................................................................. 36
netstat ...................................................................................................................................................... 36
netstat –s –p tcp ...................................................................................................................................... 37
netstat –i .................................................................................................................................................. 38
netstat –m ................................................................................................................................................ 39
InsightIQ External Network Errors ................................................................................................... 40
DNS ................................................................................................................................................. 42

NETWORKING TOOLS .......................................................................................................... 43

SUPPORTED NETWORK OPTICS AND TRANSCEIVERS .................................................. 43
REFERENCES ........................................................................................................................ 43
TAKE THE NEXT STEP ......................................................................................................... 44

4 | Isilon Network Design Considerations

© 2018 Dell Inc.
 EXECUTIVE SUMMARY
This document provides design considerations for understanding, configuring, and troubleshooting Isilon Scale-Out NAS external
networking. In a Scale-Out NAS environment, the overall network architecture must be configured to maximize the user experience.
Many factors contribute to overall network performance. This document examines network architecture design and best practices
including factors such as Latency, Flow Control, ICMP, MTU, jumbo frames, congestion, TCP/IP parameters, and IPv6.

NOTICE TO READERS
It is important to understand that the network design considerations stated in this document are based on general network design and
are provided as guidance to Isilon administrators. As these are considerations, all of these may not apply to each workload. It is
important to understand each consideration and confirm if it pertains to a specific environment.

Each network is unique, not only from a design perspective but also from a requirements and workloads perspective. Before making
any changes based on the guidance in this document, it is important to discuss modifications with the Network Engineering team.
Additionally, as a customary requirement for any major IT implementation, changes should first be tested in a lab environment that
closely mimics the workloads of the live network.

VERSION HISTORY
Version Date Comments

1.0 March 2017 Initial Rough Draft

2.0 July 2017 Updated after several reviews and posted online
3.0 November 2017 Updated after additional feedback.
Updated title from “Isilon Advanced Networking Fundamentals” to “Isilon Network Design Considerations.”
Updated the following sections with additional details:
 Link Aggregation
 Jumbo Frames
 Latency
 ICMP & MTU
New sections added:
 MTU Framesize Overhead
 Ethernet Frame
 Network Troubleshooting
3.1 December 2017 Added link to Network Stack Tuning spreadsheet
Added Multi-Chassis Link Aggregation
3.2 January 2018 Removed switch-specific configuration steps with a note for contacting manufacturer
Updated section title for Confirming Transmitted MTUs
Added OneFS commands for checking and modifying MTU
Updated Jumbo Frames section
3.3 May 2018 Updated equation for Bandwidth Delay Product

4.0 August 2018 Added the following sections:

 SyncIQ Considerations
 SmartConnect Considerations
 Access Zones Best Practices
4.1 August 2018 Minor updates based on feedback and added ‘Source-Based Routing Considerations’

4.2 September 2018 Updated links

4.3 November 2018 Added section ‘Source-Based Routing & DNS’

5 | Isilon Network Design Considerations

© 2018 Dell Inc.
 NETWORK ARCHITECTURE DESIGN
The architecture design is the core foundation of a reliable and highly available network, taking into account capacity and bandwidth.
Layered on top of the basic foundation are the many applications running on a campus network with each requiring specific features
and considerations.

For the following sections, it is important to understand the differences between Distribution and Access switches. Typically, distribution
switches perform L2/L3 connectivity while Access switches are strictly L2. The Figure below provides the representation for each.

Figure 1. Distribution and Access Switches

GENERAL NETWORK ARCHITECTURE CONSIDERATIONS

Designing a network is unique to the requirements of each enterprise data center. There is certainly not a “one size fits all” design and
not a single “good network design.” When approaching network design, it is important to use principles as a leading factor, coupled with
the enterprise requirements. The requirements must include current and future application consumption, providing the guiding factor in
major decisions. Network design is based on many concepts; the following are considerations and principles to guide the process:

 Single Points of Failure: Ensure the network design has layers of redundancy. Dependence on a single device or link relates to a
loss of resources or outages. The enterprise requirements take into account risk and budget, guiding the level of redundancy.
Redundancy should be implemented through backup paths and load sharing. If a primary link fails, traffic uses a backup path. Load
sharing creates two or more paths to the same endpoint and shares the network load. When designing access to Isilon nodes, it is
important to assume links and hardware will fail, ensuring access to the nodes survives those failures.

 Application and Protocol Traffic: Understanding the application data flow from clients to the Isilon cluster across the network allows
for resources to be allocated accordingly while minimizing latency and hops along this flow.

 Available Bandwidth: As traffic traverses the different layers of the network, the available bandwidth should not be significantly
different. Compare this available bandwidth with the workflow requirements.

 Minimizing Latency: Ensuring latency is minimal from the client endpoints to the Isilon nodes maximizes performance and
efficiency. Several steps can be taken to minimize latency, but latency should be considered throughout network design.

 Prune VLANs: It is important to limit VLANs to areas where they are applicable. Pruning unneeded VLANs is also good practice. If
unneeded VLANs are trunked further down the network, this imposes additional strain on endpoints and switches. Broadcasts are
propagated across the VLAN and impact clients.

 VLAN Hopping: VLAN hopping has two methods, switch spoofing and double tagging. Switch spoofing is when a host imitates the
behavior of a trunking switch, allowing access to other VLANs. Double tagging is a method where each packet contains two VLAN
tags, with the assigned or correct VLAN tag is empty and the second as the VLAN where access is not permitted. It is
recommended to assign the native VLAN to an ID that is not in use. Otherwise tag the native VLAN to avoid VLAN hopping,
allowing a device to access a VLAN it normally would not have access. Additionally, only allow trunk ports between trusted devices
and assign access VLANs on ports that are different from the default VLAN.

6 | Isilon Network Design Considerations

© 2018 Dell Inc.
 TRIANGLE LOOPED TOPOLOGY
This section provides best practices for Layer 2 Access network design. Although many network architectures may meet enterprise
requirements, this document takes a closer look at what is commonly referred to as the Triangle Looped Access Topology, which is the
most widely implemented architecture in enterprise data centers.

Figure 2. Triangle Looped Access Topology

The Looped Design Model extends VLANs between the aggregation switches, thus creating the looped topology. To prevent actual
loops, Spanning Tree is implemented, using Rapid PVST+ or MST. For each path, a redundant path also exists, which is blocking until
the primary path is not available. Access layer uplinks may be used to load balance VLANs. A key point to consider with the Looped
Access Topology is the utilization of the inter-switch link between the Distribution switches. The utilization must be monitored closely as
this is used to reach active services.

The Looped Triangle Access Topology supports VLAN extension and L2 adjacency across the Access layer. Through the use of STP
and dual homing, the Looped Triangle is extremely resilient. Stateful services are supported at the aggregation layer and quick
convergence with 802.1W/S.

Utilizing the Triangle Looped Topology allows for multiple Access Switches to interface with the external network of the Isilon Scale-Out
NAS environment. Each Isilon node within a cluster is part of a distributed architecture which allows each node to have similar
properties regarding data availability and management.

7 | Isilon Network Design Considerations

© 2018 Dell Inc.
 LINK AGGREGATION
In the context of the IEEE 802.1AX standard, link aggregation provides methods to combine multiple Ethernet interfaces, forming a
single link layer interface, specific to a switch or server. Therefore, link aggregation is implemented between a single switch and an
Isilon node, not across Isilon nodes.

Implementing link aggregation is neither mandatory nor is it necessary, rather it is based on workload requirements and is
recommended if a transparent failover or switch port redundancy is required.

Link aggregation assumes all links are full duplex, point to point, and at the same data rate, providing graceful recovery from link
failures. If a link fails, traffic is automatically sent to the next available link without disruption.

It is imperative to understand that link aggregation is not a substitute for a higher bandwidth link. Although link aggregation combines
multiple interfaces, applying it to multiply bandwidth by the number of interfaces for a single session is incorrect. Link aggregation
distributes traffic across links. However, a single session only utilizes a single physical link to ensure packets are delivered in order
without duplication of frames.

As part of the IEEE 802.1AX standard, the Frame Distributor does not specify a distribution algorithm across aggregated links, but
enforces that frames must be sent in order without duplication. Frame order is maintained by ensuring that all frames of a given session
are transmitted on a single link in the order that they are generated by the client. The mandate does not allow for additions or
modifications to the MAC frame, buffering, or processing to re-order frames by the Frame Distributor or Collector.

Thus, the bandwidth for a single client is not increased, but the aggregate bandwidth of all clients increases in an active/active
configuration. The aggregate bandwidth is realized when carrying multiple simultaneous sessions and may not provide a linear multiple
of each link’s data rate, as each individual session utilizes a single link.

Another factor to consider is depending on the workload, certain protocols may or may not benefit from link aggregation. Stateful
protocols, such as NFSv4 and SMBv2 benefit from link aggregation as a failover mechanism. On the contrary, SMBv3 Multichannel
automatically detects multiple links, utilizing each for maximum throughput and link resilience.

Table 1. Link Aggregation

Link Aggregation Advantages Link Aggregation Limitations
Higher aggregate bandwidth for multiple sessions. A single Provides resiliency for interface and cabling failures, but not for
session is confined to a single link. switch failures.
Link resiliency Bandwidth for a single session is not improved as a single link is
used for each session.
Ease of management with a single IP address Depending on the workload, each protocol has varying
Load balancing limitations and advantages of Link Aggregation

MULTI-CHASSIS LINK AGGREGATION

As discussed in the previous section, the IEEE 802.1AX standard does not define Link Aggregation between multiple switches and an
Isilon node. However, many vendors provide this functionality through proprietary features. Multiple switches are connected with an
Inter-Switch link or other proprietary cable and communicate via a proprietary protocol forming a virtual switch. A virtual switch is
perceived as a single switch to an Isilon node, with links terminating on a single switch. The ability to have link aggregation split with
multiple chassis provides network redundancy if a single chassis were to fail.

Each vendor has a proprietary implementation of Multi-Chassis Link Aggregation, but externally the virtual switch created is compliant
with the IEEE 802.1AX standard.

It is important to recognize that regarding bandwidth, the concepts discussed for single switch Link Aggregation still apply to Multi-
Chassis Link Aggregation. Additionally, as the multiple switches form a single virtual switch, it is important to understand what happens
if the switch hosting the control plane fails. Those effects vary by the vendor’s implementation but will impact the network redundancy
gained through Multi-Chassis Link Aggregation.

8 | Isilon Network Design Considerations

© 2018 Dell Inc.
 LATENCY, BANDWIDTH & THROUGHPUT
Maximizing overall network performance is dependent on several factors. However, the three biggest factors contributing to end-to-end
performance are latency, throughput, and bandwidth. This section focuses on these factors to maximize the Isilon user experience.

LATENCY
Latency in a packet-switched network is defined as the time from when a source endpoint sends a packet to when it is received by the
destination endpoint. Round trip latency, sometimes referred to as round-trip delay, is the amount of time for a packet to be sent from
the source endpoint to the destination endpoint, and returned from the destination to the source endpoint.

Minimal latency in any transaction is imperative for several reasons. IP endpoints, switches, and routers operate optimally without
network delays. Minimal latency between clients and an Isilon node ensures performance is not impacted. As latency increases
between two endpoints, this may lead to several issues that degrade performance heavily, depending on the application.

In order to minimize latency, it is important to measure it accurately between the endpoints. For assessing Isilon nodes, this is
measured from the clients to a specified node. The measurement could use the IP of a specific node or the SmartConnect hostname.
After configuration changes are applied that impact latency, it is important to confirm the latency has indeed decreased. When
attempting to minimize latency, consider the following points:

 Hops: Minimizing hops required between endpoints decreases latency. The implication is not to drag cables across a campus, but
the goal is to confirm if any unnecessary hops could be avoided. Minimizing hops applies at the physical level with the number of
switches between the endpoints but also applies logically to network protocols and algorithms.

 ASICs: When thinking about network hops it also important to consider the ASICs within a switch. If a packet enters through one
ASIC and exits through the other, latency could increase. If at all possible, it is recommended to keep traffic as part of the same
ASIC to minimize latency.

 Network Congestion: NFS v3, NFSv4 and SMB employ the TCP protocol. For reliability and throughput, TCP uses windowing to
adapt to varying network congestion. At peak traffic, congestion control is triggered, dropping packets, and leading TCP to utilize
smaller windows. In turn, throughput could decrease, and overall latency may increase. Minimizing network congestion ensures it
does not impact latency. It is important to architect networks that are resilient to congestion.

 Routing: Packets that pass through a router may induce additional latency. Depending on the router configuration, packets are
checked for a match against defined rules, in some cases requiring packet header modification.

 MTU Mismatch: Depending on the MTU size configuration of each hop between two endpoints, an MTU mismatch may exist.
Therefore, packets must be split to conform to upstream links, creating additional CPU overhead on routers and NICs, creating
higher processing times, and leading to additional latency.

 Firewalls: Firewalls provide protection by filtering through packets against set rules for additional steps. The filtering process
consumes time and could create further latency. Processing times are heavily dependent upon the number of rules in place. It is
good measure to ensure outdated rules are removed to minimize processing times.

BANDWIDTH & THROUGHPUT

Understanding the difference between throughput and bandwidth are important for network troubleshooting. Although these terms are
conflated at times, they are actually both unique. Bandwidth is the theoretical maximum speed a specific medium can deliver if all
factors are perfect without any form of interference. Throughput is the actual speed realized in a real-world scenario, given interference
and other environmental factors such as configuration, contention, and congestion.

The difference between these terms is important when troubleshooting. If an Isilon node supports 40 GbE, it does not necessarily mean
the throughput is 40 Gb/s. The actual throughput between a client and an Isilon node is dependent on all of the factors between the two
endpoints and may be measured with a variety of tools.

During the design phase of a data center network, it is important to ensure bandwidth is available throughout the hierarchy, eliminating
bottlenecks and ensuring consistent bandwidth. The bandwidth from the Access Switches to the Isilon nodes should be a ratio of what
is available back to the distribution and core switches. For example, if an Isilon cluster of 12 nodes has all 40 GbE connectivity to
access switches, the link from the core to distribution to access should be able to handle the throughput from the access switches.
Ideally, the link from the core to distribution to access should support roughly a bandwidth of 480 Gb (12 nodes * 40 GbE).

9 | Isilon Network Design Considerations

© 2018 Dell Inc.
 BANDWIDTH DELAY PRODUCT
Bandwidth Delay Product (BDP) is calculated to find the amount of data a network link is capable of, in bytes, which can be transmitted
on a network link at a given time. The keyword is transmitted, meaning the data is not yet acknowledged. BDP takes into consideration
the bandwidth of the data link and the latency on that link, in terms of a round-trip delay.

The amount of data that can be transmitted across a link is vital to understanding Transmission Control Protocol (TCP) performance.
Achieving maximum TCP throughput requires that data must be sent in quantities large enough before waiting for a confirmation
message from the receiver, which acknowledges the successful receipt of data. The successful receipt of the data is part of the TCP
connection flow. The diagram below explains the steps of a TCP connection and where BDP is applicable:

Figure 3. Transmission Control Protocol Message Flow

In the diagram above, four states are highlighted during a TCP connection. The following summarizes each state:

1. TCP Handshake – Establishes the TCP connection through an SYN, SYN/ACK, ACK

2. Data transmitted to the server. BDP is the maximum amount of data that can be sent at this
step.

3. Data acknowledged by Server

4. TCP Connection Close Sequence – Socket closure is initiated by either side

Once the BDP rate is calculated, the TCP stack is tuned for the maximum throughput, which is discussed in the next section. The BDP
is calculated by multiplying the bandwidth of the network link (bits/second) by the round-trip time (seconds).

For example, a link with a bandwidth of 1 Gigabit per second and a 1 millisecond round trip time, would be calculated as:
Bandwidth * RTT = 1 Gigabit per second * 1 millisecond =
1,000,000,000 bits per second * 0.001 seconds = 1,000,000 bits = 0.125 MB

Thus, 0.125 MB may be sent per TCP message to the server.

10 | Isilon Network Design Considerations

© 2018 Dell Inc.
 ISILON NETWORK STACK TUNING
Once the BDP is calculated and understood, these findings can be applied to modifying the TCP stack on the Isilon cluster. All Isilon
clusters do not require TCP stack tuning. Only alter the TCP stack for a needed workflow improvement. The majority of Isilon
environments do not need TCP tuning. Before applying any TCP changes, ensure the network is clean and reliable by performing basic
checks for excessive retransmits, duplicate or fragmented packets, and broken pipes.

Isilon OneFs is built on FreeBSD. An Isilon cluster is composed of nodes with a distributed architecture, and each node provides
external network connectivity. Adapting the TCP stack to bandwidth, latency, and MTU requires tuning to ensure the cluster provides
optimal throughput.

In the previous section, BDP was explained in depth and how it is the amount of data that can be sent across a single TCP message
flow. Although the link supports the BDP that is calculated, the OneFS system buffer must be able to hold the full BDP. Otherwise, TCP
transmission failures may occur. If the buffer does not accept all of the data of a single BDP, the acknowledgment is not sent, creating a
delay, and the workload performance is degraded.

The OneFS network stack must be tuned to ensure on inbound, the full BDP is accepted, and on outbound, it must be retained for a
possible retransmission. Prior to modifying the TCP stack, it is important to measure the current I/O performance and then again after
implementing changes. As discussed earlier in this document, the tuning below is only guidance and should be tested in a lab
environment before modifying a production network.

The spreadsheet below provides the necessary TCP stack changes based on the bandwidth, latency, and MTU. The changes below
must be implemented in the order below and all together on all nodes. Modifying only some variables could lead to unknown results.
After making changes, it is important to measure performance again.

Note: The snippet below is only for representation. It is imperative to input the calculated bandwidth, latency, and MTU specific to each
environment.

Figure 4. Isilon TCP Network Stack Tuning

Enter values below
Available bandwidth (Gb/s) 40
Latency (ms) 5
MTU 1500

Bandwidth delay product (MB) 25

Recommended
Sys Controls
value Default value
102400 kern.ipc.maxsockbuf 104857600 2097152
25600 net.inet.tcp.recvspace 26214400 131072
25600 net.inet.tcp.sendspace 26214400 131072
51200 net.inet.tcp.sendbuf_max 52428800 262144
51200 net.inet.tcp.recvbuf_max 52428800 262144
16 net.inet.tcp.sendbuf_inc 16384 8192
32 net.inet.tcp.recvbuf_inc 32768 16384
net.inet.tcp.rfc1323 1 1
net.inet.tcp.sack.enable 1 1
net.inet.tcp.mssdflt 1448 512
net.inet.tcp.path_mtu_discovery 1 1

Download the Isilon Network Stack Tuning spreadsheet at the following link:
http://www.emc.com/collateral/tool/h164888-isilon-onefs-network-stack-tuning.xlsm

11 | Isilon Network Design Considerations

© 2018 Dell Inc.
 ETHERNET FLOW CONTROL
Under certain conditions, packets sent from the source to the destination can overwhelm the destination endpoint. The destination is
not able to process all of the packets at the rate that they are sent, leading to retransmits or dropped packets. Most scenarios have a
fast source endpoint and a slower destination endpoint; this could be due to processing power or several source endpoints interacting
with a single destination. Flow Control is implemented to manage the rate of data transfer between these IP endpoints, providing an
option for the destination to control the data rate, and further ensuring the destination is capable of processing all of the packets from
the source.

The IEEEs 802.3x standard defines an Ethernet Flow Control mechanism at the data link layer. It specifies a pause flow control
mechanism through MAC Control frames in full duplex link segments. For Flow Control to be successfully implemented, it must be
configured throughout the network hops that the source and destination endpoints communicate through. Otherwise, the pause flow
control frames will not be recognized and will be dropped.

Isilon OneFS listens for pause frames but does not send them, meaning it is only applicable when an Isilon node is the source. OneFS
recognizes pause frames from the destination.

Most networks today do not send pause frames, but certain devices still send them. In particular, Cisco Nexus Switches with the Fabric
Extender Modules have been known to send pause frames.

If the network or cluster performance does not seem optimal, it is easy to check for pause frames on an Isilon cluster. To check for
pause frames received by an Isilon cluster, execute the following command from the shell:
isi_for_array -a <cluster name> sysctl dev | grep pause

Check for any values greater than zero. In the example, below, the cluster has not received any pause frames. If values greater than
zero are printed consistently, Flow Control should be considered.

Figure 5. Checking for Pause Frames

If pause frames are reported, it is important to discuss these findings with the Network Engineering team before making any changes.
As mentioned above, changes must be implemented across the network, ensuring all devices recognize a pause frame. Please contact
the switch manufacturer’s support teams or account representative for specific steps and caveats for implementing flow control before
proceeding.

12 | Isilon Network Design Considerations

© 2018 Dell Inc.
 SYNCIQ CONSIDERATIONS
Isilon SyncIQ provides asynchronous data replication for disaster recovery and business continuance, allowing failover and failback
between clusters. It is configurable for either complete cluster replication or only for specific directories. Within an Isilon cluster, all
nodes can participate in replication. After an initial SyncIQ replication, only changed data blocks are copied minimizing network
bandwidth and resource utilization on clusters.

This section provides considerations for SyncIQ pertaining to external network connectivity. For more information on SyncIQ, see the
Best Practices for Data Replication with EMC Isilon SyncIQ white paper.

SYNCIQ DISASTER RECOVERY WITH SMARTCONNECT

This section describes best practices for disaster recovery planning with OneFS SmartConnect.

Dedicated static SmartConnect zones are required for SyncIQ replication traffic. As with any static SmartConnect zone, the dedicated
replication zone requires one IP address for each active logical interface. For example, in the case of two active physical interfaces,
10gige-1 and 10gige-2, requiring two IP addresses. However, if these are combined with link aggregation, interface 10gige-agg-1 only
requires one IP address. Source-restrict all SyncIQ jobs to use the dedicated static SmartConnect zone on the source cluster, and
repeat the same on the target cluster.

By restricting SyncIQ replication jobs to a dedicated static SmartConnect Zone, replication traffic may be assigned to specific nodes,
reducing the impact of SyncIQ jobs on user or client I/O. The replication traffic is directed without reconfiguring or modifying the
interfaces participating in the SmartConnect zone.

For example, consider a data ingest cluster for a sports television network. The cluster must ingest large amounts of data recorded in
4K video format. The data must be active immediately, and the cluster must store the data for extended periods of time. The sports
television network administrators want to keep data ingestion and data archiving separate, to maximize performance. The sports
television network purchased two types of nodes: H500s for ingesting data, and A200s for the long-term archive. Due to the extensive
size of the the data set, SyncIQ jobs replicating the data to the disaster recovery site, have a significant amount of work to do on each
pass. The front-end interfaces are saturated on the H500 nodes for either ingesting data or performing immediate data retrieval. The
CPUs of those nodes must not be effected by the SyncIQ jobs. By using a separate static SmartConnect pool, the network
administrators can force all SyncIQ traffic to leave only the A200 nodes and provide maximum throughput on the H500 nodes.

REPLICATION TRAFFIC OVER DEDICATED WAN LINKS

Depending on the network topology and configuration, in certain cases Isilon SyncIQ data may be sent across a dedicated WAN link
segregated from client traffic. Under these circumstances, the recommended option is utilizing a different subnet on the Isilon cluster for
replication traffic, separated from the subnet for user data access.

ISILON ONEFS PORTS

Isilon OneFS uses a number of TCP and UDP ports, which are documented in the Security Configuration Guide available at the
following link: https://community.emc.com/docs/DOC-57599

13 | Isilon Network Design Considerations

© 2018 Dell Inc.
 SMARTCONNECT CONSIDERATIONS
This section provides considerations for using the Isilon SmartConnect load-balancing service. The general IP routing principles are the
same with or without SmartConnect.

SmartConnect acts as a DNS delegation server to return IP addresses for SmartConnect zones, generally for load-balancing
connections to the cluster. The IP traffic involved is a four-way transaction shown in the figure below.

Figure 6. SmartConnect DNS Delegation Steps

In the figure above, the arrows indicate the following steps:

1. [Blue Arrow – Step 1] The client makes a DNS request for sc-zone.domain.com by sending a DNS request packet to the site
DNS server.

2. [Green Arrow – Step 2] The site DNS server has a delegation record for sc-zone.domain.com and sends a DNS request to
the defined nameserver address in the delegation record, the SmartConnect service (SmartConnect Service IP Address).

3. [Orange Arrow – Step 3] The cluster node hosting the SmartConnect Service IP (SSIP) for this zone receives the request,
calculates the IP address to assign based on the configured connection policy for the pool in question (such as round robin),
and sends a DNS response packet to the site DNS server.

4. [Red Arrow – Step 4] The site DNS server sends the response back to the client.

SMARTCONNECT NETWORK HIERARCHY

As SmartConnect subnets and pools are defined it is important to understand the SmartConnect hierarchy, as displayed in the following
figure:

Figure 7. Isilon External Network Hierarchy

Throughout the network design phase, consider that a single SSIP is defined per subnet. However, under each subnet, pools are
defined, and each pool will have a unique SmartConnect Zone Name. It is important to recognize that multiple pools leads to multiple
SmartConnect Zones utilizing a single SSIP. As shown in the diagram above, a DNS provider is defined per Groupnet, which is a
feature in OneFS 8.0 and newer releases. In releases before 8.0, a DNS per Groupnet was not supported.

14 | Isilon Network Design Considerations

© 2018 Dell Inc.
 LOAD BALANCING
SmartConnect load balances incoming network connections across SmartConnect Zones composed of nodes, network interfaces, and
pools. The load balancing policies are Round Robin, Connection Count, CPU Utilization, and Network Throughput. The most common
load balancing policies are Round Robin and Connection Count, but this may not apply to all workloads. It is important to understand
whether the front-end connections are being evenly distributed, either in count or by bandwidth. Front-end connection distribution may
be monitored with InsightIQ or the webUI. It is important to understand how each Load Balancing Policy functions, and testing it in a lab
environment prior to a production roll-out, as each workload is unique. The table below lists suggested policies based on the workflow,
but these are general suggestions, and may not always be applicable.

Generally speaking, starting with Round Robin is recommended for a new implementation or if the workload is not clearly defined. As
the workload is further defined and based on the Round Robin experience, another policy can be tested in a lab environment.

Table 2. Suggested SmartConnect Load Balancing Policies

Load Balancing Workload
Policy General Few Clients with Many Persistent NFS & Many Transitory NFS Automounts
or Other Extensive Usage SMB Connections Connections (HTTP, FTP) or UNC Paths
Round Robin     
Connection Count*    
CPU Utilization*
Network Throughput*

*Metrics are gathered every 5 seconds for CPU Utilization and every 10 seconds for Connection Count and Network Throughput. In
cases where many connections are created at the same time, these metrics may not be accurate, creating an imbalance across nodes.

As discussed previously, the above policies mapping to workloads are general guidelines. Each environment is unique with distinct
requirements. It is recommended to confirm the best load balancing policy in a lab environment which closely mimics the production
environment. For additional details on the description of each SmartConnect load balancing policy, refer to the Isilon SmartConnect
White Paper.

STATIC VERSUS DYNAMIC IP ADDRESS ALLOCATION

After a groupnet and subnet are defined in OneFS, the next step is configuring an IP address pool and assigning interfaces to
participate in this pool. The lowest IP address of the pool is assigned to the lowest Logical Node Number (LNN) from the selected
interfaces, subsequently for the second lowest IP address and LNN.

Once the IP address pool is defined, under the ‘SmartConnect Advanced’ Section, an ‘Allocation Method’ may be selected. By default,
this option is grayed out as ‘Static’ if a SmartConnect Advanced license is not installed. If a SmartConnect Advanced license is
installed, the default ‘Allocation Method’ is still ‘Static’, but ‘Dynamic’ may also be selected.

The Static Allocation Method assigns a single persistent IP address to each interface selected in the pool, leaving additional IP
addresses in the pool unassigned if the number of IP addresses is greater than interfaces. In the event a node or interface becomes
unavailable, this IP address does not move to another node or interface. Additionally, when the node or interface becomes unavailable,
it is removed from the SmartConnect Zone and new connections will not be assigned to the node. Once the node is available again,
SmartConnect adds it back into the zone and assigns new connections.

On the contrary, the Dynamic Allocation Method splits all available IP addresses in the pool across all selected interfaces. OneFS
attempts to assign the IP addresses evenly if at all possible, but if the interface to IP address ratio is not an integer value, a single
interface may have more IP addresses than another.

DYNAMIC FAILOVER
Combined with the Dynamic Allocation Method, Dynamic Failover provides high-availability by transparently migrating IP addresses to
another node when an interface is not available. If a node becomes unavailable, all of the IP addresses it was hosting are re-allocated
across the new set of available nodes in accordance with the configured failover load balancing policy. The default IP address failover
policy is round-robin, which evenly distributes IP addresses from the unavailable node across available nodes. As the IP address
remains consistent, irrespective of which node it resides on, this results in an transparent failover to the client, providing seamless high
availability.

The other available IP address failover policies are the same as the initial client connection balancing policies, i.e., connection count,
throughput, or CPU usage. In most scenarios, round-robin is not only the best option, but also the most common. However, the other

15 | Isilon Network Design Considerations

© 2018 Dell Inc.
 failover policies are available for specific workflows. As mentioned previously, with the initial load balancing policy, test the IP failover
policies in a lab environment to find the best option for a specific workflow.

DYNAMIC FAILOVER EXAMPLES

In order to understand Dynamic Failover, the following examples illustrate how IP addresses move during a failover.

The examples below illustrate the concepts of how the IP address quantity impacts user experience during a failover and these are the
guidelines to use when determining IP address quantity.

DYNAMIC ZONE WITH 1 IP ADDRESS PER NODE

This example considers a four node cluster with one network connection per node and one dynamic SmartConnect zone with only four
IP addresses. One IP address will be assigned to each node, as shown in the following figure:

Figure 8. Dynamic Zone – 4 Node Cluster with 1 IP Address Per Node

In this scenario, 150 clients are actively connected to each node over NFS using a round-robin connection policy. Most NFSv3 mounted
clients perform a nslookup only the first time that they mount, never performing another nslookup to check for an updated IP address. If
the IP address changes, the NFSv3 clients have a stale mount and retain that IP address.

Suppose that one of the nodes fails, as shown in the following figure:

Figure 9. Dynamic Zone – 4 Node Cluster with 1 IP Address Per Node – 1 Node Offline

A SmartConnect Zone with a dynamic allocation strategy immediately hot-moves the one IP address on the failed node to one of the
other three nodes in the cluster. It sends out a number of gratuitous address resolution protocol (ARP) requests to the connected
switch, so that client I/O continues uninterrupted.

Although all four IP addresses are still online, two of them—and 300 clients—are now connected to one node. In practice,
SmartConnect can fail only one IP to one other place, and one IP address and 150 clients are already connected to each of the other
nodes. The failover process means that a failed node has just doubled the load on one of the three remaining nodes while not

16 | Isilon Network Design Considerations

© 2018 Dell Inc.
 disrupting the other two nodes. Therefore, this process results in declining client performance, but not equally. The goal of any scale-
out NAS solution must be consistency. To double the I/O on one node and not on another is inconsistent.

DYNAMIC ZONE WITH 3 IP ADDRESS PER NODE

Dynamic SmartConnect zones require a greater number of IP addresses than the number of nodes at a minimum to handle failover
behavior. In the example below, the formula used to calculate the number of IP addresses required is N*(N-1), where ‘N’ is the number
of nodes. The formula is used for illustration purposes only to demonstrate how IP addresses, and in turn clients, move from one node
to another, and how this could potentially lead to an imbalance across nodes. Every workflow and cluster is unique, and this formula is
not applicable to every scenario.

This example considers the same four node cluster as the previous example, but now following the rule of N*(N-1). In this case 4*(3-1)
= 12, equaling three IPs per node, as shown in the figure below.

Figure 10. Dynamic Zone – 4 Node Cluster with 3 IP Addresses Per Node

When the same failure event as the previous example occurs, the three IP addresses are spread over all the other nodes in that
SmartConnect zone. This failover results in each remaining node having 200 clients and four IP addresses. Although performance may
degrade to a certain degree, it may not be as drastic as the failure in the first scenario, and the experience is consistent for all users, as
shown in the following figure.

Figure 11. Dynamic Zone – 4 Node Cluster with 3 IP Addresses Per Node – 1 Node Offline

17 | Isilon Network Design Considerations

© 2018 Dell Inc.
 PROTOCOLS AND SMARTCONNECT ALLOCATION METHODS
A common concern during an Isilon configuration is selecting between Static and Dynamic Allocation methods. The requirement for
Dynamic Failover depends heavily on the protocol in use, workflow, and overall high-availability design requirements. Stateful versus
stateless protocols combined with the allocation method, impact the failover experience. Certain workflows require minimal downtime,
or the overarching IT requirements dictate IP address persistence. This section provides guidance on failover behavior based on the
protocol.

Client access protocols are either stateful or stateless. Stateful protocols are defined by the client/server relationship having a session
state for each open file. Failing over IP addresses to other nodes for these types of workflows means that the client assumes that the
session state information was carried over. Session state information for each file is not shared among Isilon nodes. On the contrary,
stateless protocols are generally accepting of failover without session state information being maintained, except for locks.

SMB
Typically, SMB performs best in static zones. In certain workflows, SMB is preferred in a dynamic zone, because IP address
consistency is required. It may not only be a workflow requirement, but could also be an IT administrative dependence. SMB actually
works well with dynamic zones, but it is essential to understand the protocol limitations. SMB preserves complex state information per
session on the server side. If a connection is lost and a new connection is established with dynamic failover to another node, the new
node may not be able to continue the session where the previous one had left off. If the SMB workflow is primarily reads, or heavier on
the read side, the impact of a dynamic failover will not be as drastic, as the client can re-open the file and continue reading. Conversely,
if an SMB workflow is primarily writes, the state information is lost, and the writes could be lost, possibly leading to file corruption.
Hence, in most cases, static zones are suggested for SMB, but again it is workflow dependent. Prior to a major implementation, it is
recommend to test the workflow in a lab environment, understanding limitations and the best option for a specific workflow.

NFS
The NFSv2 and NFSv3 protocols are stateless, and in almost all cases perform best in a dynamic zone. The client does not rely on
writes unless commits have been acknowledged by the server, enabling NFS to failover dynamically from one node to another.

The NFSv4 protocol introduced state making it a better fit for static zones in most cases, as it expects the server to maintain session
state information. However, OneFS 8.0 introduced session-state information across multiple nodes for NFSv4, making dynamic pools
the better option. Additionally, most mountd daemons currently still behave in a v3 manner, where if the IP address it’s connected to
becomes unavailable, this results in a stale mount. In this case, the client does not attempt a new nslookup and connect to a different
node.

Again, as mentioned above, test the workflow in a lab environment to understand limitations and the best option for a specific workflow.

HDFS
The requirements for HDFS pools have been updated with the introduction of new OneFS features and as HDFS environments have
evolved. During the design phases of HDFS pools, several factors must be considered. The use of static versus dynamic pools are
impacted, by the following:

 Use of OneFS racks if needed

 Node Pools – is the cluster a single heterogeneous node type or do different Node Pools exist

 Availability of IP addresses

The factors above coupled with the workflow requirements determine the pool implementation. Please reference the HDFS Pool Usage
and Assignments section in the EMC Isilon Best Practices Guide for Hadoop Data Storage for additional details and considerations with
HDFS pool implementations.

SUGGESTED ZONES BY PROTOCOL

The table below lists the suggested IP allocation strategies for SmartConnect Advanced by the protocol. As noted, these are
suggested, and the actual zone type is dependent on the workflow requirements, as discussed above.

18 | Isilon Network Design Considerations

© 2018 Dell Inc.
 Table 3. Suggested Protocols and Zone Types

Protocol Protocol Category Suggested Zone Type

NFSv2 (not supported in OneFS 7.2 and above) Stateless Dynamic

NFSv3 Stateless Dynamic
NFSv4 Stateful Dynamic or Static – Depending
on mountd daemon and OneFS
version. Refer to NFS section
above.
SMBv1 Stateful
Dynamic or Static – Refer to
SMBv2 / SMBv2.1 Stateful
SMB section above
SMBv3 Multi-Channel Stateful
FTP Stateful Static
SFTP / SSH Stateful Static
HDFS Stateful – Protocol is tolerant of failures Refer to EMC Isilon Best
Practices Guide for Hadoop
Data Storage
HTTP / HTTPS Stateless Static
SyncIQ Stateful Static Required

Note: SynqIQ can add nodes while a sync job is running.

IP ADDRESS QUANTIFICATION
This section provides guidance for determining the number of IP addresses required for a new cluster implementation. The guidance
provided below does not apply to all clusters, and is provided as a reference for the process and considerations during a new cluster
implementation.

During the process of implementing a new cluster and building the network topology, consider the following:

 Calculate the number of IP addresses that are needed based on future cluster size, not the initial cluster size.

 Do not share a subnet with other application servers. If more IP addresses are required, and the range is full, re-addressing an
entire cluster and then moving it into a new VLAN is disruptive. These complications are prevented with proper planning.

 Static IP pools require one IP address for each logical interface that will be in the pool. Each node provides 2 interfaces for
external networking. If Link Aggregation is not configured, this would require 2*N IP addresses for a static pool.

 1 IP address for each SmartConnect Service IP (SSIP)

 For optimal load-balancing, during a node failure, IP pools with the Dynamic Allocation Method require the number of IP
addresses at a minimum of the node count and a maximum of the client count. For example, a 12-node SmartConnect zone
and 50 clients, would have a minimum of 12 and maximum of 50 IP addresses. In many larger configurations, defining an IP
address per client is not feasible, and in those cases the optimal number of IP addresses is workflow dependent and based on
lab testing. In the previous examples, N*(N-1) is used to calculate the number of IP addresses, where N is the number of
nodes that will participate in the pool. For larger clusters, this formula may not be feasible due to the sheer number of IP
addresses. Determining the number of IP addresses within a Dynamic Allocation pool varies depending on the workflow, node
count, and the estimated number of clients that would be in a failover event.

In previous OneFS releases, a greater IP address quantity was recommended considering the typical cluster size and the workload a
single node could handle during a failover. As nodes become unavailable, all the traffic hosted on that node is moved to another node
with typically the same resources, which could lead to a degraded end-user experience. Isilon nodes are now in the 6th generation, and
this is no longer a concern. Each node does have limitations, and those must be considered when determining the number of IP
addresses and failover events creating additional overhead. Additionally, as OneFS releases have progressed, so has the typical
cluster size, making it difficult to maintain the N*(N-1) formula with larger clusters.

From a load-balancing perspective, for dynamic pools, it is ideal, although optional, that all the interfaces have the same number of IP
addresses, whenever possible. It is important to note that in addition to the points above, consider the workflow and failover
requirements set by IT administrators.

19 | Isilon Network Design Considerations

© 2018 Dell Inc.
 SMARTCONNECT NODE SUSPENSION
OneFS SmartConnect provides an option to administratively remove a node from a SmartConnect Zone during a planned outage.
Planned outages could be hardware replacement or maintenance activity. Suspending a node is particularly useful when a node is the
lowest Logical Node Number (LNN) and is hosting the SmartConnect Service IP (SSIP).

Once a node is suspended, SmartConnect prevents new client connections to the node. If the node is part of a dynamic zone, IP
addresses are not assigned to this node in a suspended state. Suspending a node ensures that client access remains consistent. After
the node is suspended, client connections can be monitored and allowed to gradually drop-off before a reboot or power down.

A node is suspended from the OneFS CLI or web interface. From the Isilon CLI, the command is:
isi network pools --sc-suspend-node <groupnet.subnet.pool> <node ID>

Alternatively, from the web interface, click “Suspend Nodes” under the ‘Pool,’ as displayed in the following figure:

Figure 12. SmartConnect Node Suspension

After a node is suspended, new connections are not created. Prior to rebooting or shutting the node down, confirm all client connections
have dropped by monitoring the web interface under the “Client Connections” tab from the “Cluster Overview” page. Also, clients may
have to be manually booted from the node if they have static SMB connections with applications that maintain connections.

SMARTCONNECT AND REVERSE DNS

SmartConnect does not provide reverse lookups. In most cases, it is recommended that Isilon SmartConnect Service IP addresses and
SmartConnect Zone names, do not have reverse DNS entries, also known as pointer (PTR) records.

In certain environments where PTR records may be required, this results in the creation of many PTR entries, as Isilon SmartConnect
pools could have hundreds of IP addresses. In scenarios where PTR records are required, each time an additional IP address is added
to a SmartConnect pool, DNS changes are necessary to keep the environment consistent.

Creating reverse DNS entries for the SmartConnect Service IP’s Host [address, or A] record is acceptable if the SmartConnect Service
IP is referenced only with an A record in one DNS domain.

DNS DELEGATION BEST PRACTICES

This section describes DNS delegation best practices for Isilon clusters.

DELEGATE TO ADDRESS (A) RECORDS, NOT TO IP ADDRESSES

The SmartConnect service IP address on an Isilon cluster, in most cases, should be created in DNS as an address (A) record, also
called a host entry. An A record maps a URL such as www.dell.com to its corresponding IP address. Delegating to an A record is
provides simplicity during a failover. Only a single DNS A record must be updated. All other name server delegations can be left alone.
In many enterprises, it is easier to have an A record updated than to update a name server record, because of the perceived complexity
of the process.

20 | Isilon Network Design Considerations

© 2018 Dell Inc.
 USE ONE NAME SERVER RECORD FOR EACH SMARTCONNECT ZONE NAME OR ALIAS

One delegation for each SmartConnect zone name or each SmartConnect zone alias on a cluster is recommended. This method
permits failover of only a portion of the cluster's workflow—one SmartConnect zone—without affecting any other zones. This method is
useful for scenarios such as testing disaster recovery failover and moving workflows between data centers.

It is not recommended to create a single delegation for each cluster and then create the SmartConnect zones as sub-records of that
delegation. Using this method would enable Isilon administrators to change, create, or modify their SmartConnect zones and zone
names as needed without involving a DNS team. The concern with this method is it causes failover operations that involve the entire
cluster and affects the entire workflow, not just the impacted SmartConnect zone.

SMARTCONNECT IN ISOLATED NETWORK ENVIRONMENTS

SmartConnect is, effectively, a limited implementation of a custom DNS server: It answers only for the SmartConnect zone names or
aliases configured on it. In order to use SmartConnect in an isolated network environment where no DNS infrastructure is available
(such as a DMZ), configure the client systems using the SmartConnect service IP address as the primary DNS server. Configuring the
client systems this way ensures the following:

 Requests to connect to Isilon clusters with SmartConnect zone names will succeed

 The isolated network benefits from SmartConnect features, such as load-balancing and rerouting traffic to prevent unavailable
nodes, will work as expected in a typical, non-isolated deployment.

 It is important to recognize that Isilon OneFS is not a full DNS server, hence it will only answer for SmartConnect Zones.

The following commands show how to simulate and test a configuration that uses the SmartConnect service IP address as the primary
DNS server.

C:\>nslookup

Default Server: 10.123.17.60

Address: 10.123.17.60

> isi01-s0.domain.com

Server: [10.123.17.60]

Address: 10.123.17.60

Name: isi01-s0.domain.com

Address: 10.123.17.64

> isi01-s0.domain.com

Server: [10.123.17.60]

Address: 10.123.17.60

Name: isi01-s0.domain.com

Address: 10.123.17.63

21 | Isilon Network Design Considerations

© 2018 Dell Inc.
 SMARTCONNECT DNS, SUBNET, AND POOL DESIGN
This section provides a starting point for planning SmartConnect DNS, subnet, and SmartConnect pool layouts that meet the needs of
most new cluster implementations with a SmartConnect Advanced License.

SMARTCONNECT ZONE NAMING

It is recommended to ensure SmartConnect Zones are named according to relevant details for clarity and simple recognition. For
example, names should be composed of some or all of the following variables:

 Cluster Name: Active Directory (AD) uses the cluster name as the AD machine account name. For example, when a cluster
named isi01 joins Active Directory, isi01 is the cluster’s machine account name. Using the cluster/machine account name in all
DNS entries simplifies cluster administration and troubleshooting.

 IP Allocation Strategy: Each SmartConnect zone has an IP allocation strategy set to static or dynamic. The allocation strategy
is allocated in the zone name, for example, by using “d” for dynamic and or “s” for static.

 SmartConnect Pool ID: Each SmartConnect pool has a unique name or number that identifies it. By default, the first pool
called on a cluster is pool0, the second is pool1, and so on. These identifiers are recommended to be part of the zone name.

 SSIP: Use the SSIP in the zone name to indicate a SmartConnect Service IP zone.

The variables above together form a SmartConnect zone name. For example: isi01-s0.domain.com

The name includes the cluster name (isi01), the allocation strategy of the zone (“s” for static), and the number of the pool (pool0).

For example, a cluster with three pools:

 pool0: Static for client I/O for stateful protocols

 pool1: Dynamic for client I/O for stateless protocols

 pool2: Static for Backup and Replication

Based on the SmartConnect zone, pool, and the cluster information, the following is a sample DNS layout for the cluster named 'isi01':

 isi01-ssip.domain.com in [A] to 10.x.y.z

 isi01-s0.domain.com in [NS] to isi01-ssip.domain.com

 isi01-d1.domain.com in [NS] to isi01-ssip.domain.com

 isi01-s2.domain.com in [NS] to isi01-ssip.domain.com

SMARTCONNECT WITH MULTIPLE NODE POOLS OR TYPES

From a client or application perspective, the goals for all scale-out NAS deployments are consistency and availability. Consistency, in
this context, implies that every time a client connects, whether that client is an application server or a user opening their home directory,
the same level of performance is provided. Dell EMC offers a number of different Isilon node types with varied performance profiles.

Many factors determine performance in network-attached storage. In an Isilon cluster, key components are the front-end performance,
which consists of the network card, CPU, and memory in the node that is serving the relevant data protocol, and the back-end
performance, which, in this case, is the disk tier or pool where the data resides. In the context of SmartConnect configuration, creating a
connection pool that spans across different node performance levels is not recommended. For example, a pool with Isilon F800 nodes
and A200 nodes would provide significantly varying protocol performance. It is imperative to understand how the nodes within a
connection pool impact client experience.

22 | Isilon Network Design Considerations

© 2018 Dell Inc.
 WHERE THE SMARTCONNECT SERVICE IP RUNS
The Isilon clustered compute and storage platform has no single point of failure. However, the SmartConnect DNS service must be
active on only one node at any time, per subnet. The SmartConnect Service IP resides on the node with the lowest node ID that has an
interface in the given subnet, not necessarily on the node with the lowest Logical Node Number (LNN) in the cluster.

To illustrate how this works, suppose that an existing four-node cluster is refreshed with four new nodes. Assume that the cluster has
only one configured subnet, all the nodes are on the network, and that there are sufficient IP addresses to handle the refresh. The first
step in the cluster refresh, is to add the new nodes with the existing nodes, temporarily creating an eight node cluster. Next, the original
four nodes are SmartFailed. The cluster is then composed of the four new nodes with the original data set.

As the administrators perform the refresh, they check the current configuration using the isi config command, with the status
advanced command, as shown in the following example:
isi config
>status advanced

The SmartConnect service continues to run throughout the process as the existing nodes are refreshed. The following example
illustrates where the SmartConnect service runs at each step in the refresh process.

Once the four new nodes are added to the cluster, based on the existing naming convention, they are automatically named
clustername-5, clustername-6, clustername-7, and clustername-8. At this point, the Node IDs and LNNs are displayed in the following
table:

Table 4. 8-Node Cluster Configuration – Before SmartFail

Logical Node Number (LNN) NodeID Node Name New or Original Node
1 1 clustername-1 Original
2 2 clustername-2 Original
3 3 clustername-3 Original
4 4 clustername-4 Original
5 5 clustername-5 New
6 6 clustername-6 New
7 7 clustername-7 New
8 8 clustername-8 New
The SmartConnect service always runs on the node with the lowest node ID, at this point, NodeID 1 is mapping to LNN 1.

Next, the original nodes are removed using SmartFail. The updated Node IDs and LNNs are displayed in the following table:

Table 5. 4-Node Cluster Configuration – After SmartFail

Logical Node Number (LNN) NodeID Node Name New or Original Node
1 5 clustername-5 New
2 6 clustername-6 New
3 7 clustername-7 New
4 8 clustername-8 New
The SmartConnect service always runs on the node with the lowest node ID, at this point, NodeID 5 is mapping to LNN 1.

Keeping the naming convention consistent, the administrators re-name the new nodes, formerly clustername-5, clustername-6,
clustername-7, and clustername-8, to clustername-1, clustername-2, clustername-3, and clustername-4, respectively. The updated
Node IDs and LNNs remain the same, but map to a different Node Name, as displayed in the following table:

Table 6. 4-Node Cluster Configuration – After Re-Name

Logical Node Number (LNN) NodeID Node Name New or Original Node
1 5 clustername-1 New
2 6 clustername-2 New
3 7 clustername-3 New
4 8 clustername-4 New
The SmartConnect service always runs on the node with the lowest node ID, at this point, NodeID 5 is mapping to LNN 1.

If LNN 1 is offline for maintenance, the SmartConnect service migrates to LNN 2, because LNN 2 has the next lowest NodeID number,
6.

23 | Isilon Network Design Considerations

© 2018 Dell Inc.
 OTHER SMARTCONNECT CONSIDERATIONS
During SmartConnect configuration, consider the following points:

 It is recommended to disable client DNS caching, when possible. To handle client requests properly, SmartConnect requires
that clients use the latest DNS entries. If clients cache SmartConnect DNS information, they could connect to incorrect
SmartConnect zone names. In this event, SmartConnect might not appear to be functioning correctly.

 If traffic is traversing firewalls, ensure that the appropriate ports are open. For example, if UDP port 53 is opened, also ensure
TCP port 53 is opened.

 Certain clients perform DNS caching and might not connect to the node with the lowest load if they make multiple connections
within the lifetime of the cached address. For example, this issue occurs on Mac OS X for certain client configurations.

 In order to successfully distribute IP addresses, the OneFS SmartConnect DNS delegation server answers DNS queries with a
time-to-live (TTL) of 0 so that the answer is not cached. Certain DNS servers, most particularly Windows Server 2003, 2008,
2012, and 2016, will update the value to one second. If many clients are requesting an address within the same second, this
will cause all of them to receive the same address. If this occurs frequently, consider a different DNS server, such as bind.

 The site DNS servers must be able to communicate with the node that is currently hosting the SmartConnect service.

 Site DNS servers might not exist in the regular local subnets, or in any of the subnets that clients occupy. To enable the
SmartConnect lookup process, ensure that the DNS servers use a consistent route to the cluster and back. If the site DNS
server sends a lookup request that arrives through one local subnet on the cluster, but the configured cluster routing causes
the response to be sent through a different subnet, it’s likely that the packet will be dropped and the lookup will fail. The
solutions and considerations for SmartConnect are similar to the client scenarios. Additionally, the DNS server might benefit
from a static route to the subnet that contains the SSIP address or addresses.

 SmartConnect makes it possible for different nodes to have different default routes, but this is fundamentally determined by
connectivity. SmartConnect enables administrators to define multiple gateways, with 1 gateway per subnet. Each gateway is
assigned a priority when it is defined. On any node, SmartConnect attempts to use the highest priority gateway—the gateway
that has the lowest number—that has an available functioning interface in a subnet that contains the gateway address.

24 | Isilon Network Design Considerations

© 2018 Dell Inc.
 ETHERNET, MTU, & IP OVERHEAD
A Maximum Transmission Unit (MTU) is the largest packet size or frame, which may be sent along a link. The MTU is specified in
octets and is utilized by TCP to determine the maximum size of a packet per transmission. A large MTU size provides less overhead as
packet headers and acknowledgments are not consuming space and bandwidth. However, this could lead to retransmissions or drops if
a hop does not support it. On the contrary, a small MTU size is not as efficient as overhead increases with packet headers and
acknowledgments.

Generally speaking, the MTU across the internet is 1500 bytes. As such, most devices limit packet size to roughly 1472 bytes, allowing
for additional overhead and remaining under the 1500 byte limit. Additional overhead may be added as the packet goes through
different hops. The IEEE 802.3 standard also specifies 1500 bytes as the standard payload.

ETHERNET PACKET
An Ethernet frame carries a payload of data and is carried by an Ethernet packet. The frame could be IPv4 or IPv6 and TCP or UDP.
The IEEE 802.3 standard defines the structure of each packet. As a packet traverses different layers, the structure is modified
accordingly. In the diagram below, the structure is displayed as it would traverse the wire, or Layer 1. Dissecting how a packet is
structured on the wire lends to an understanding of how the packet overhead is impacted and all of the other components required to
send a payload.

Figure 13. Ethernet Packet

An Ethernet packet on the wire at Layer 1 is composed of the following fields:

 Interpacket Gap: Serves as a gap between each frame, similar to a spacer. The Interpacket gap is only part of Layer 1. The
field originates from a time when hubs were common, and collisions were more commonplace.

 Preamble: Composed of alternating 1 and 0 bits for receiver clock synchronization. The Preamble is only part of Layer 1.

 Start Frame Delimiter: Identifies the start of an Ethernet frame.

 Destination MAC: Contains the MAC address of the destination station for which the data is intended.

 Source MAC: Contains the MAC address of the sending station.

 VLAN 802.1Q: Optional field utilized if a VLAN is identified.

 Type: Also known as the EtherType field, this defines the type of protocol that is encapsulated in the payload. In the example
above, it is an Ethernet II Frame, the most widely accepted type.

 Payload: Spans from 46 to 1500 bytes and contains user data. If it is smaller than 46 bytes, blank values are entered to bring
this up to 46 bytes as it is the minimum value. The Payload consists of protocol data for TCP, UDP or RTP and IPv4 or IPv6.
The next section explains the Payload field in greater depth.

25 | Isilon Network Design Considerations

© 2018 Dell Inc.
  CRC: Cyclic Redundancy Check is part of the Frame Check Sequence (FCS) to detect errors within the frame. The CRC code
should result in a zero if the data does not contain any errors.

ETHERNET PAYLOAD
The Ethernet payload varies based on the type of data it is carrying. It is a combination of either TCP, UDP, or RTP header combined
with an IPv4 or IPv6 header, and most importantly the actual payload which contains the data that is being sent. The fields within the
payload are displayed in the figure below.

Figure 14. Ethernet Payload Options

As shown in the figure above, the amount of actual data sent within an Ethernet Frame is dependent upon the number of bytes
consumed by the other fields. There are other options are available which are not listed here. For example, Linux hosts automatically
add a timestamp to the TCP stack, adding 12 bytes.

JUMBO FRAMES
Jumbo frames are Ethernet frames where the MTU is greater than the standard 1500 bytes and a maximum of 9000 bytes. The larger
MTU size provides greater efficiency as less overhead and fewer acknowledgments are sent across devices, drastically reducing
interrupt load on endpoints. Jumbo frames are recommended for most workloads as the amount of data sent per message is far
greater, reducing processing times and maximizing efficiency. While the general assumption is that Jumbo frames provide performance
advantages for all workloads, it is important to measure results in a lab environment simulating a specific workload to ensure
performance enhancements.

For Jumbo frames to take advantage of the greater efficiencies, they must be enabled end-to-end on all hops between endpoints.
Otherwise, the MTU could be lowered through PMTUD or packets could be fragmented. The fragmentation and reassembly impact the
CPU performance of each hop, which impacts the overall latency.

For example, if a client is set to an MTU of 1500 bytes while other hops are set to 9000 bytes, transmission along the path will most
likely set to 1500 bytes using PMTUD, unless other options are configured.

Jumbo frames utilize the same Ethernet packet structure described in the previous section. However, the difference is the size of the
data within the payload. As the byte consumption of the other components within the frame remain the same, each packet contains
more data with the same overhead. A Jumbo frame Ethernet payload is displayed in the following figure:

26 | Isilon Network Design Considerations

© 2018 Dell Inc.
 Figure 15. Jumbo Ethernet Payload

IP PACKET OVERHEAD
Isilon nodes utilize 10 and 40 GbE NICs for front-end networking. In order to maximize throughput on these high bandwidth links,
Jumbo frames are recommended for greater throughput. Standard 1500 byte and Jumbo 9000 byte packets are formed with the same
packet structure at Layer 1 with the only difference pertaining to the actual data payload size. Although the overhead is identical for
standard and Jumbo packets, the ratio of the data to the overhead varies significantly.

For every payload sent to Layer 1 on the wire, the following fields are required:
Interpacket Gap / Preamble / Start Frame Delimiter / Destination MAC / Source MAC / Type / CRC

In bytes, this translates to:

12 + 7 + 1 + 6 + 6 + 2 + 4 = 38 bytes

Hence, regardless of the payload fields or size, every payload requires an additional 38 bytes to be sent. It is important to note that this
does not take into account the optional VLAN tag which requires an additional 4 bytes. The following sections provide examples of
packet overhead based on the payload fields.

EXAMPLE 1: STANDARD 1500 BYTE PAYLOAD – IPV4 / TCP

IPv4 and TCP headers consume the following bytes:
20 bytes (IPv4) + 20 bytes (TCP)= 40 bytes

If the payload headers consume 40 bytes, the data field for a standard 1500 byte payload consumes:
1500 – 40 = 1460 bytes

Therefore, a standard 1500 byte packet with IPv4 and TCP headers results in a data to Ethernet frame percentage as follows:
(Data Bytes) / (Total Ethernet Frame Bytes) = (1500 – 40) / (1500 + 38) = 1460/1538 = .949 => 94.9%

EXAMPLE 2: JUMBO 9000 BYTE PAYLOAD – IPV4 / TCP

A standard 9000 byte payload that contains IPv4 and TCP headers consumes the following bytes:
20 bytes (IPv4) + 20 bytes (TCP)= 40 bytes

If the payload headers consume 40 bytes, the data can field consumes:
9000 – 40 = 8960 bytes

Therefore, a standard 1500 byte packet with IPv4 and TCP headers results in a data to Ethernet frame percentage as follows:
(Data Bytes) / (Total Ethernet Frame Bytes) = (9000 – 40) / (9000 + 38) = 8960/9038 = .991 => 99.1%

27 | Isilon Network Design Considerations

© 2018 Dell Inc.
 EXAMPLE 3: STANDARD 1500 BYTE PAYLOAD – IPV4 / TCP / LINUX TIMESTAMP
Linux automaticlly inserts the timestamp within the payload. A standard 1500 byte payload that contains IPv4, TCP, and timestamp
headers consumes the following bytes:
20 bytes (IPv4) + 20 bytes (TCP) + 12 bytes (timestamp) = 52 bytes

If the payload headers consume 40 bytes, the data field consumes:

1500 – 52 = 1448 bytes

Therefore, a standard 1500 byte packet with IPv4 and TCP headers results in a data to Ethernet frame percentage as follows:
(Data Bytes) / (Total Ethernet Frame Bytes) = (1500 – 52) / (1500 + 38) = 1448/1538 = .941 => 94.1%

EXAMPLE 4: JUMBO 9000 BYTE PAYLOAD – IPV4 / TCP / LINUX TIMESTAMP

Linux automaticlly inserts the timestamp within the payload. A standard 9000 byte payload that contains IPv4, TCP and timestamp
headers consumes the following bytes:
20 bytes (IPv4) + 20 bytes (TCP) + 12 bytes (timestamp) = 52 bytes

If the payload headers consume 40 bytes, the data field consumes:

9000 – 52 = 8948 bytes

Therefore, a standard 1500 byte packet with IPv4 and TCP headers results in a data to Ethernet frame percentage as follows:
(Data Bytes) / (Total Ethernet Frame Bytes) = (9000 – 52) / (9000 + 38) = 8948/9038 = .990 => 99.0%

DATA PAYLOAD TO ETHERNET FRAME EFFECIENCY

Utilizing the calculations above, the table below provides additional examples of the amount of data that is sent per Ethernet frame for
standard and Jumbo frames.

Table 7. Data Payload to Ethernet Frame Percentage

Data to Ethernet Frame Percentage
Packet Type
Standard Frame Jumbo Frame
IPv4 / TCP 94.93% 99.14%
IPv4 / TCP / Linux Timestamp 94.15% 99.00%
IPv4 / TCP / Linux Timestamp / VLAN 93.90% 98.96%
IPv6 / TCP 93.63% 98.92%
IPv6 / TCP / Linux Timestamp 92.85% 98.78%
IPv6 / TCP / Linux Timestamp / VLAN 92.59% 98.74%
IPv4 / UDP 95.71% 99.27%
IPv4 / UDP / Linux Timestamp 94.93% 99.14%
IPv4 / UDP / Linux Timestamp / VLAN 94.67% 99.09%
IPv6 / UDP 94.41% 99.05%
IPv6 / UDP / Linux Timestamp 93.63% 98.92%
IPv6 / UDP / Linux Timestamp / VLAN 93.37% 98.87%
Note: NFS v2 is UDP. NFS v3 and v4 are TCP. SMB is TCP.

As shown in the table above, Jumbo frames deliver between 98%-99% efficiency depending on the packet type. The efficiencies are
only maximized when all hops from the client endpoint to an Isilon node support Jumbo frames. Otherwise, packets may be fragmented
leading to additional processing overhead on devices or PMTUD finding the lowest MTU along the path. Therefore, Jumbo frames are
recommended for optimal performance. However, it is important to recognize that each workload environment is unique and measuring
performance enhancements in a lab are recommended prior to a production network update.

28 | Isilon Network Design Considerations

© 2018 Dell Inc.
 ICMP AND MTU WITH ISILON
Network devices employ Internet Control Message Protocol (ICMP) to gather communications related information. ICMP is capable of
sending error messages but also delivers operational information. Ping and TraceRoute both send ICMP messages to provide
connectivity information including latency and network hops.

Most devices have a default MTU that is configurable and remains at the defined value. Isilon OneFS determines MTU size specific to
each transaction. After the initial TCP handshake, the Isilon node sends an ICMP message for Path MTU Discovery (PMTUD), RFC
1191, gathering the maximum supported MTU size. If for any reason ICMP is disabled, or PMTUD is not supported, this causes OneFS
to default the MTU size to 536 bytes, which typically leads to performance degradation.

ISILON ONEFS MTU COMMANDS

To check the current configured MTU, enter the following command:
isi networks subnets list –v

To modify the MTU, use the isi command with the following context:
isi network subnets modify groupnet0.subnet1 --mtu=1500 --gateway=198.162.100.10 --gateway-priority=1

CONFIRMING TRANSMITTED MTU

Manually checking a permitted MTU size ensures a configured size is transmitted. The ping command is used to confirm if an MTU size
can be transmitted. It is recommended to start with the largest MTU and work down to find the limit.

For example, to check if an MTU size of 8900 bytes is transmitted to an endpoint, from the OneFS CLI, use the following command:
ping –s 8900 –D <IP Address>. The ‘-s’ specifies the packet size and the ‘-D’ specifies the not to fragment the packet.

If the ping is successful, the MTU size is transmitted across. If the ping is unsuccessful, gradually lower the MTU size until it is
successfully transmitted. Confirm the MTU size can be transmitted from both endpoints.

OneFS is based on FreeBSD. FreeBSD also has options for gradually increasing the MTU size by performing a ‘sweeping ping’ using
the –g option. For more information on ping options in FreeBSD, access the FreeBSD manual at the following link:
https://www.freebsd.org/cgi/man.cgi?ping(8)

29 | Isilon Network Design Considerations

© 2018 Dell Inc.
 ACCESS ZONES BEST PRACTICES
When Access Zones are configured, a root based path must be defined to segment data into the appropriate Access Zone and enable
the data to be compartmentalized. Access Zones carve out access to an Isilon cluster creating boundaries for multi-tenancy or multi-
protocol. They permit or deny access to areas of the cluster. At the Access Zone level, authentication providers are also provisioned.

SYSTEM ZONE
When an Isilon cluster is first configured, the System Zone is created by default. The System Zone should only be used for
management as a best practice. In certain special cases, some protocols require the system zone, but generally speaking, all protocol
traffic should be moved to an Access Zone. If nothing else, NFS and SMB should have protocol specific Access Zones.

Moving client traffic to Access Zones ensures the System Zone is only used for management and accessed by administrators. Access
Zones provide greater security as administration, and file access is limited to a subset of the cluster, rather than the entire cluster.

ROOT BASED PATH

SmartConnect zones map to Access Zones, which map to a Root Based Path. When an Access Zone is defined, a Root Based Path
must be defined. Best practice is to use the cluster name, a numerical Access Zone number, and a directory. For example, Access
Zone 1 maps to /ifs/clustername/az1/<data directories>, Access Zone 2 maps to /ifs/clustername/az2/<data directories>. A Root Based
Path with this delineation, provides data separation, Multi-Tenancy, maintains the Unified Permission model and makes SyncIQ failover
and failbacks easier.

Generally speaking, the best practice to remove all data access from the default System Zone. Otherwise, this leads to complications in
the future as the cluster grows and additional teams or workflows are added. Further, as mentioned above, create a subdirectory under
the Access Zone, rather than using the root of the zone, as this makes migration and disaster recovery simpler. It is preferred not to
have an overlap of Root Based Paths unless it is required for a specific workflow. Overlap is supported in 8.0 and newer releases
through the CLI.

In the figure below, as Cluster 1 fails over to Cluster 2, the directory structure remains consistent, easily identifying where the files
originated from. This delineation also ensures clients have the same directory structure after a failover. Once the IP address is updated
in DNS, the failover is transparent to clients. As more clusters are brought together with SyncIQ, this makes it easier to manage data,
understanding where it originated from and provides seamless disaster recovery.

Figure 16. Importance of Root Based Path Best Practices

Root Based Paths may also be based on protocol. As an example, protocols are matched with a Root Based Path in the following table:

Table 8. Protocol Specific Access Zones

Protocol Root Based Path
NFS Access /ifs/cls1/AZ1/nfs
SMB Access /ifs/cls1/AZ2/smb
NFS / SMB / HDFS /ifs/cls1/AZ3/mp

30 | Isilon Network Design Considerations

© 2018 Dell Inc.
 SOURCE-BASED ROUTING CONSIDERATIONS
Source-Based Routing (SBR) with Isilon OneFS is discussed in the Isilon OneFS 8.1.0 External Network Connectivity Guide. This
section clarifies how SBR functions. The naming convention suggests that SBR is routing packets based on a source IP address.
However, SBR is actually a mechanism to dynamically create per-subnet default routes. The router used as this gateway is derived
from the subnet configuration. Gateways must be defined for each subnet. For example, consider a cluster with subnets A, B, and C,
as illustrated in the following figure:

Figure 17. Source-Based Routing

In the example above, each gateway has a defined priority. If SBR is not configured, the highest priority gateway, i.e. gateway with the
lowest value which is reachable, is used as the default route. Once SBR is enabled, when traffic arrives from a subnet that is not
reachable via the default gateway, firewall rules are added. As OneFS is FreeBSD based, these are added through ipfw. In the example
above, the following ipfw rules are provisioned:

If src-ip is in subnetA and dst-ip is not in (subnetA,B,C) set next-hop to gatewayA

If src-ip is in subnetB and dst-ip is not in (subnetA,B,C) set next-hop to gatewayB
If src-ip is in subnetC and dst-ip is not in (subnetA,B,C) set next-hop to gatewayC

The process of adding ipfw rules is stateless and essentially translates to per-subnet default routes. SBR is entirely dependent on the
source IP address that is sending traffic to the cluster. If a session is initiated from the source subnet, the ipfw rule is created. The
session must be initiated from the source subnet, otherwise the ipfw rule is not created. If the cluster has not received traffic that
originated from a subnet that is not reachable via the default gateway, OneFS will transmit traffic it originates through the default
gateway. Given how SBR creates per-subnet default routes, consider the following:

 A subnet setting of 0.0.0.0 is not supported and is severely problematic, as OneFS does not support RIP, RARP, or CDP.

 The default gateway is the path for all traffic intended for clients that are not on the local subnet and not covered by a routing
table entry. Utilizing SBR does not negate the requirement for a default gateway, as SBR in effect overrides the default
gateway, but not static routes.

 Static routes are an option when the cluster originates the traffic and the route is not accessible via the default gateway. As
mentioned above, static routes are prioritized over source-based routing rules.

31 | Isilon Network Design Considerations

© 2018 Dell Inc.
 SOURCE-BASED ROUTING & DNS
As discussed earlier in this paper, it’s important to understand the path a specific session traverses throughout the network hierarchy. If
SBR is configured on a cluster, this will also impact how the cluster creates sessions with other hosts, such as a DNS server.

In certain environments, Isilon clusters with SBR enabled and multiple SmartConnect SIP (SSIP) addresses, have experienced
excessive latency with DNS responses. As mentioned previously in this paper, keeping latency minimal is imperative through any
transaction and the delayed DNS responses could impact DNS dependent workflows. The prior section explained how SBR
dynamically assigns a gateway. In this instance, the route to the DNS server is changed as the session originated on a different
interface based on the SSIP being addressed.

In order to prevent the additional latency with DNS responses, when SBR is enabled with multiple SSIP addresses, consider the
following:

 If a single Access Zone is configured to have multiple SmartConnect zones and multiple subnets with SBR enabled, it is
recommended to have a single SSIP.

 If a cluster is using a single DNS server, it is recommended to use a single SSIP.

 If multiple Access Zones are required within a single groupnet, then a single SSIP is recommended.

32 | Isilon Network Design Considerations

© 2018 Dell Inc.
 IPv6
Although Internet Protocol version 4 (IPv4) is the most common version of IP today, Internet Protocol version 6 (IPv6) is the newest
version and ultimately replaces IPv4. IPv4 addresses were completely allocated to specific geographic regions in 2011. IPv6 uses 128-
bit addresses supporting 340 undecillion addresses. For those unfamiliar with an undecillion, this translates to 340 times 10 to the 36th
power possible IP addresses.

WHY IPV6?
IPv6 brings innovation and takes connectivity to a new level with enhanced user experiences.

SECURITY
IPv6 supports IPSEC inherently with encryption and integrity checks. Additionally, the Secure Neighbor Discovery (SEND) protocol
provides cryptographic confirmation of host identity, minimizing hostname based attacks like Address Resolution Protocol (ARP)
poisoning, leading to devices placing more trust in connections.

EFFICIENCY
IPv6’s large address space means many devices no longer require NAT translation as previously with IPv4, making routers far more
efficient. Overall data transmission is faster and simplified as the need for checking packet integrity is eliminated.

MULTICAST
IPv6 supports multicast rather than broadcast, allowing media streams to be sent to multiple destinations simultaneously leading to
bandwidth savings.

QUALITY OF SERVICE
QoS implementation is far simplified in IPv6 with a new packet header. The IPv6 header contains a new field, Flow Label, which
identifies packets belonging to the same flow. The Flow Label associates packets from a specific host and head to a particular
destination.

33 | Isilon Network Design Considerations

© 2018 Dell Inc.
 IPV6 ADDRESSING
IPv6’s address structure is defined by the IETF as part of RFC 3513 and provides many of the advantages discussed above over IPv4.
At first glance, it is evident an IPv6 address, looks nothing like an IPv4 address. IPv4 addresses are composed of four numerical octets,
ranging from 0 to 255, separated by periods, forming a 32-bit address. IPv6 addresses are 128 bits and consisting of a series of eight
segments, separated by a colon. Each segment is a 4-character hexadecimal number, ranging from 0000 to FFFF, representing 16 bits
each, totaling to the 128 bits.

Figure 18. IPv6 Address

For display purposes, an IPv6 address may be presented without leading zeros. For example, an IPv6 address as 2001 : 0DC8:
E004 : 0001 : 0000 : 0000 : 0000 : F00A could be displayed as 2001 : DC8: E004 : 1 : 0 : 0 : 0 : F00A.

The address may be further reduced by removing consecutive fields of zeros and replacing with a double-colon. The double-colon can
only be used once in an address. The address above becomes 2001 : DC8 : E004 : 1 :: F00A.

IPv6 offers the following address types:

 Unicast: one-to-one – Single Address to Single Interface

 Anycast: one-to-nearest – Assigned to a group of interfaces, with packets being delivered only to a single (nearest) interface

 Multicast: one-to-many – Assigned to a group of interfaces, and is typically delivered across multiple hosts.

An IPv6 Unicast address is composed of the Global Routing Prefix, Subnet ID, and the Interface Identifier. The Global Routing Prefix is
the network ID or prefix of the address for routing. The Subnet ID is similar to the netmask in IPv4 but is not part of the IP address in
IPv6. Finally, the Interface ID is a unique identifier for a particular interface. For Ethernet networks, the Ethernet MAC address (48 bits)
may be used for the Interface Identifier, by inserting 16 additional bits, forming what is referred to as an EUI-64 address.

Figure 19. IPv6 Unicast Address Format

34 | Isilon Network Design Considerations

© 2018 Dell Inc.
 IPV6 HEADER
An IPv6 header is simplified in comparison to IPv4, minimizing complexity and making the header easier to process for all devices.
Efficiency was one of the focus points with IPv6 from the onset, which is brought to light with the faster processing of IPv6 headers. The
figure below displays the format of an IPv6 header.

Figure 20. IPv6 Header

The table below defines the fields of an IPv6 header.

Table 9. IPv6 Header Fields

Field Description Length
Version Specifies if the packet is IPv4 or IPv6 4 Bits
Traffic Class Similar to an IPv4 Type of Service field and includes support for Differentiated Services Code Point 8 Bits
(DSCP) providing congestion control.
Flow Label Provides the ability to track certain traffic flows at the network layer for QoS management. 20 Bits
Payload Length Similar to the IPv4 ‘Length’ field – Provides length of the data portion 16 Bits
Next Header Similar to the IPv4 ‘Protocol’ field – Provides what to expect after the basic header, including 8 Bits
options for a TCP or UDP header
Hop Limit Similar to the IPv4 ‘Time to Live’ field – Provides the maximum number of hops 8 Bits

IPV6 TO IPV4 TRANSLATION

Connecting IPv6 and IPv4 remains a challenge with the slow migration to IPv6 and support for legacy devices requiring IPv4. The three
top options available to facilitate IPv4 and IPv6 communication are dual-stack networks, tunneling, and translation.

For Service Providers to deliver IPv6, they utilize translation technologies. The two major translation technologies are the Network
Address Translation IPv6 to IPv4 (NAT64) and Stateless IP/ICMP Translation (SIIT). NAT64 is similar to the IPv4 Network Address
Translation but is specific to IPv6. SIIT is capable of replacing IPv4 and IPv6 as part of the translation.

CONFIGURING ISILON ONEFS FOR IPV6

Implementing IPv6 on an Isilon cluster is a simple process as IPv6 and IPv4 are supported as a dual stack. In order to provision an IPv6
subnet with Isilon, follow these steps:

1. Select an existing Groupnet or create a new one

2. Enter DNS servers and add a Subnet

3. Create Subnet by selecting IPv6 in the IP Family

4. Create network address pool and assign interfaces

35 | Isilon Network Design Considerations

© 2018 Dell Inc.
 NETWORK TROUBLESHOOTING
This section provides steps for assessing and troubleshooting network issues with generally available utilities.

NETSTAT
Netstat, short for network statistics, is a utility built into most Windows and Linux clients. It provides an array of statistics on current
ports, routing, IP stats for transport layer protocols, and serves as a forensic tool to link processes with network performance while
digging deeper into the current network status. Netstat bundles several actions into a single command with different options available.
As Netstat is multi-platform, the syntax is similar across platforms with slight variations.

NETSTAT
In its standard form without any arguments, netstat provides an overview of the current network status broken out by each connection
or socket. Each column displays the following:

 Proto: Protocol of the active connection. The protocol could be TCP or UDP and has a ‘4’ or ‘6’ associated specifying if it is
IPv4 or IPv6, respectively.

 Recv-Q and Send-Q: Value of the receiving and sending queue in bytes. Non-zero values specify the number of bytes in the
queue that are awaiting to be processed. The preferred value is zero. If several connections have non-zero values, this implies
something is causing processing to be delayed.

 Local Address and Foreign Address: Lists the hosts and ports the sockets are connected with. Some of these are local
connections to the host itself.

 State: Displays the current state of the TCP connection, based on the TCP protocol. As UDP is a stateless protocol, the ‘State’
column will be blank for UDP connections. The most common TCP states include:

o Listen: Waiting for an external device to establish a connection

o Established: Ready for communication on this connection

o Close Wait: The remote machine has closed the connection, but the local device has not closed the connection yet.

o Time Wait: The local machine is waiting for a period of time after sending an ACK to close a connection.

For more information about the states of a TCP connection, see RFC 793.

Figure 21. netstat

Netstat reveals a lot of information about the current status of network connections, and it also provides information for a more thorough
forensic analysis. While reviewing the output from netstat, some of the scenarios can be generalized, like the following:

 Recv-Q has a value greater than zero but is in a ‘Close Wait’ state. This indicates that these sockets should be torn down but
are hanging. If several sockets are in this state, it could imply the application is having difficulty tearing down the connection
and may warrant additional investigation.

 Connections that have localhost as the ‘Local’ and ‘Foreign’ address denote an internal process using the TCP stack to
communicate. These connections are not concerning and are standard practice.

36 | Isilon Network Design Considerations

© 2018 Dell Inc.
 NETSTAT –S –P TCP
Netstat offers several options, but the ‘-s’ provides statistics by protocol and ‘-p’ displays the net to media tables. These options reveal
current health and the ‘tcp’ limits it to the TCP protocol. Below, is a sample output of this command with the areas to examine
highlighted in red.

Figure 22. netstat –s –p tcp

The fields highlighted in red above must be reviewed as a ratio of the total packets that are transmitted and received as a percentage.
Additionally, these statistics should be monitored for sudden increments. As a rule of thumb, under 1% is not concerning but this also
depends on the workload. The fields highlighted above provide the following:

 Retransmitted Packets: Packets that are retransmitted consume network bandwidth and could be the reason for further
investigation. However, examining the percentage is critical. In this case, 249379 out of 235829612 were retransmitted, which
is 0.105%.

 Duplicate Acknowledgements: High latency between endpoints may lead to duplicate acknowledgments, but the ratio must be
examined. In this case, it is 0.419%. This number varies depending on the workload.

 Out of Order Packets: Out of order packets are placed in order by TCP before presenting to the application layer, which
impacts the CPU and overall stack performance as additional effort is involved in analyzing the packets. Performance is
impacted the most when packets arrive out of order with a significant time gap, or a number of packets are out of order. The
ratio, in this case, is 0.197%, which is negligible.

37 | Isilon Network Design Considerations

© 2018 Dell Inc.
 NETSTAT –I
Another option for netstat is the –i option, which is the interface display, listing cumulative statistics for total packets transferred, errors,
MTU, and collisions by interface. As netstat –i lists all available interfaces, the back-end, and front-end interfaces are displayed. A
sample output of netstat –i is shown with the –h option, making it easier to interpret, in the following figure:

Figure 23. netstat –i

From the output above, netstat –i, lists the following columns:

 Name: Network Interface Card (NIC) name. Loopback interfaces are listed as ‘lo0,’ and ‘ib’ specifies InfiniBand.

 MTU: Lists the MTU specified for the interface.

 Network: Specifies the network associated with the interface.

 Address: MAC address of the interface.

 Ipkts: Input packets are the total number of packets received by this interface.

 Ierrs: Input errors are the number of errors reported by the interface when processing the ‘Ipkts.' These errors include
malformed packets, buffer space limitation, checksum errors, errors generated by media, and resource limitation errors. Media
errors are errors specific to the physical layer, such as the NIC, connection, cabling, or switch port. Resource limitation errors
are generated at peak traffic when interface resources are exceeded by usage.

 Idrop: Input drops are the number of packets that were received, but not processed and consequently dropped on the wire.
Dropped packets typically occur during heavy load.

 Opkts: Output packets are the total number of packets transmitted by this interface

 Oerrs: Output errors are the number of errors reported by the interface when processing the ‘Opkts.' Examples of errors
include the output queue reaching limits or an issue with the host.

 Coll: Collisions are the number of packet collisions that are reported. Collisions typically occur during a duplex mismatch or
during high network utilization.

In general, errors and dropped packets require closer examination. However, as noted in the previous netstat section, the percentage of
errors and dropped packets are the main factor. The following are some of the points to consider for further analysis:

 ‘Ierrs’ should typically be less than 1% of the total ‘Ipkts.' If greater than 1%, check ‘netstat –m’ for buffer issues and consider
increasing the receive buffers. Prior to implementing changes on a production system, buffer changes should be tested in a
lab environment. Refer to the Isilon Network Stack Tuning Section for additional details.

 ‘Oerrs’ should typically be less than 1% of the total ‘Opkts.’ If greater than 1%, it could be a result of network saturation,
otherwise consider increasing the send queue size.

38 | Isilon Network Design Considerations

© 2018 Dell Inc.
  The ratio of ‘Coll’ to ‘Opkts,' should typically be less than 10%. If greater than 10%, it could be a result of high network
utilization.

NETSTAT –M
The netstat –m option displays the current status of network memory requests as mbuf clusters. Netstat –m is a powerful option for a
complete forensic analysis when one of the other netstat commands mentioned above raises concern. If mbufs are exhausted, the
node cannot accept any additional network traffic.

Figure 24. netstat –m

The netstat –m output provides information in regards to available and used mbufs. The area highlighted in red, confirms if any memory
requests have been denied. In the example above, a quick glance at this area reveals that no requests have been denied.

For more information on netstat options, visit the FreeBSD manual netstat page at
https://www.freebsd.org/cgi/man.cgi?query=netstat&manpath=SuSE+Linux/i386+11.3

39 | Isilon Network Design Considerations

© 2018 Dell Inc.
 INSIGHTIQ EXTERNAL NETWORK ERRORS
Isilon InsightIQ reports external network errors under the “Performance Reporting” tab when the “Network Performance Report” is
selected. A sample of this output is displayed in the following figure:

Figure 25. InsightIQ Network Errors

InsightIQ gathers network errors using the output from ‘netstat –i’ on external interfaces only. The total of the ‘Ierrs’ and ‘Oerrs’ is
combined and displayed in the graph. Refer to the previous section for interpreting the output from ‘netstat –i.’

In order to find the exact interfaces the errors, sort the graph by ‘Node,’ ‘Direction,’ and ‘Interface,’ as shown in the following figures:

Figure 26. InsightIQ External Network Errors by Node

40 | Isilon Network Design Considerations

© 2018 Dell Inc.
 Figure 27. InsightIQ External Network Errors by Direction

Figure 28. InsightIQ External Network Errors by Interface

From the figures above, it is concluded that the external network errors reside on the interface ‘7/10gige-1’ of Node 7, on the input or
receive side. Further analysis must be performed on this interface to conclude the root cause. Refer to the ‘netstat –i’ section in this
paper for the next troubleshooting steps.

41 | Isilon Network Design Considerations

© 2018 Dell Inc.
 DNS
DNS or Domain Name Service resolves hostnames to IP addresses. Most enterprises have a local DNS to resolve hostnames
managed by them, and then a public internet DNS resolves external hostnames. Troubleshooting DNS is performed with the utilities,
‘nslookup’ or ‘dig.’ Both provide similar information, however, ‘dig’ is more detailed. In this section, the usage of ‘dig’ is explored.

Figure 29. dig dell.com

The ‘dig’ command displays results in the following sections:

 Header: The Header provides the version of ‘dig’, options the ‘dig’ command used and the flags that are displayed.

 Question Section: The Question Section displays the original input provided to the command. In the case above, dell.com was
queried. The default is to query the DNS A record. Other options are available for querying MX and NS records.

 Answer Section: The Answer Section is the output received by dig from the DNS server queried.

 Authority Section: The Authority Section lists the available Name Servers of dell.com. They have the authority to respond to
this query.

 Additional Section: The Additional Section resolves the hostnames from the Authority Section to IP addresses.

 Stats Section: The footer at the end of the query is referred to as the Stats Section. It displays the when, where, and time the
query consumed.

Dig supports an array of options. The most common options include a reverse look-up using ‘dig –x [IP address]’ to find a host name.
The other is to specify a DNS server to query using ‘dig @[dns server] [hostname].’

For the complete list of dig options, please refer to the FreeBSD manual page:
https://www.freebsd.org/cgi/man.cgi?query=dig&sektion=1&manpath=FreeBSD%209.1-RELEASE

42 | Isilon Network Design Considerations

© 2018 Dell Inc.
 NETWORKING TOOLS
The following tools help troubleshooting or lab configurations for testing:

Brocade Virtual Router: http://www.brocade.com/en/products-services/software-networking/network-functions-virtualization/vrouter.html

SUPPORTED NETWORK OPTICS AND TRANSCEIVERS

For information about the optics and transceivers supported by Isilon nodes, see the Isilon Supportability and Compatibility Guide.

REFERENCES
Cisco Nexus 5000 Series Configuration Guide:
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli/CLIConfigurationGuide/QoS.html#pgfId
-1138522

Cisco Catalyst 6500 Best Practices:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/best/practices/recommendations.html#wp1039880

Brocade Switching Configuration Guide:

http://www.brocade.com/content/html/en/configuration-guide/NI_05800a_SWITCHING/GUID-8B890954-65A7-41DA-8E10-
7A88DFDA687C.html

Arista Product Documentation:

https://www.arista.com/en/support/product-documentation

Configuration of Jumbo MTU on Nexus 5000 and 7000 Series:

http://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/112080-config-mtu-nexus.html

IEEE Standards
https://standards.ieee.org/findstds/standard/802.1AX-2008.html

SMBv3 Multi-Channel:
https://blogs.technet.microsoft.com/josebda/2012/06/28/the-basics-of-smb-multichannel-a-feature-of-windows-server-2012-and-smb-3-
0/

OneFS 8.1.0 Documentation - Isilon Info Hub

OneFS 8.1 Web Administration Guide

OneFS 8.1 CLI Administration Guide

OneFS 8.1 API Reference

EMC Isilon SmartConnect White Paper

EMC Isilon OneFS and IsilonSD 8.1 Technical Specifications Guide

OneFS 8.1 Security Configuration Guide

RFCs:
http://www.faqs.org/rfcs/rfc1812.html
http://www.faqs.org/rfcs/rfc1122.html
http://www.faqs.org/rfcs/rfc1123.html
https://tools.ietf.org/html/rfc3971
https://tools.ietf.org/html/rfc792
https://tools.ietf.org/html/rfc3513

43 | Isilon Network Design Considerations

© 2018 Dell Inc.
 TAKE THE NEXT STEP
Contact your Dell EMC sales representative or authorized reseller to learn more about how Isilon scale-out NAS storage solutions can
benefit your organization.

Shop Dell EMC Isilon to compare features and get more information.

© 2017 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC and other trademarks are
trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective
owners. Reference Number: H16103
© 2017 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC and other trademarks are
Learn more
trademarks about
of Dell Inc. or its Dell Contact
subsidiaries. Other a Dell
trademarks mayEMC Expertof their respective
be trademarks View more resources Join the conversation
EMC
owners. Isilon Number:
Reference solutions H16103 with #DellEMCStorage

*Legal footnotes minietus idi dolore, qui velibus andisquas min rest auditiorum que el ipsandant re volupit aliquisserio il eum fuga.
44 |AtIsilon
© 2018 faccus Network
Dell Inc. dolor siDesign
nem. Nam,
or its subsidiaries. Considerations
to occus
All Rights eicDell,
Reserved. to quas acestru
EMC and mquias andebis
other trademarks etur of
are trademarks sitDell
fugitassin
Inc. or nestrum.
©
its 2018 Dell Other
subsidiaries. Inc. trademarks may be trademarks of their respective owners. Reference Number: H16463
Learn more about
Dell EMC Isilon Contact a Dell EMC Expert