Data Center Structured Cabling
Data Center Structured Cabling
Storage Switching
PoP
SAN
Server Area
Server Area
PoP
Server Area
Server Area
SAN
PoP
Figure 1.1
Functional Areas of the Data Center | Drawing ZA-3580
Storage Zone Network Efficiency
This area of the data center provides the back-end Data centers have seen significant growth in size and num-
connection to data. This area contains many types of bers in the past few years and should continue to see signifi-
storage devices. The protocols used to communicate in this cant growth in the future as networks continue to evolve
area are Fibre Channel Ethernet and small computer system and move toward 100 Gigabit Ethernet. Due to the consid-
interface (SCSI). erable growth in data centers, there is a need to have simple,
efficient cabling solutions that maximize space and facilitate
Regardless of the type of data center to be implemented, reduced installation time and costs. Preterminated solutions
there are three fundamental issues, or concerns, that are often the preferred solution as they provide higher fiber
should be addressed when evaluating each area of the density, reduced installation time and the ability to easily
data center infrastructure: facilitate moves, adds and changes (MACs).
DESTINATION SOURCE
SOF
Figure 2.2
Ethernet Frame Format | Drawing ZA-3675
The IEEE 802.3z and 802.3ae task force groups released Optical Receiver Optical Transmitter
MTP® Connector MTP Connector
standards for Gigabit Ethernet and 10 Gigabit Ethernet in
1998 and 2002, respectively. The primary 1G and 10G Rx
12 Fiber Position 1
Tx
Fibre Channel
Figure 2.4
Fibre Channel is a high-performance, low latency, duplex Parallel Optics for 40G Ethernet | Drawing ZA-3299
fiber serial link application with data rates of 1 Gb/s, 2 Gb/s,
4 Gb/s, 8 Gb/s, 10 Gb/s and 16 Gb/s. It provides a very
reliable form of communication that guarantees delivery from the server host bus adapter (HBA) to the SAN Z
of information. The Fibre Channel T11 technical commit- director to the SAN storage. Similar to Ethernet, OM3
tees are responsible for developing transmitting guidance. and OM4 fibers are the dominant fibers and media type
Fibre Channel is used in the data center to transmit data used in the SAN network. Fibre Channel networks to date
TABLE 2.1
1G: Multimode 1G: Single-mode
have exclusively used optical media for the backbone as Electronic Engineer’s (IEEEs) Data Center Bridging
well as the interconnect into the electronics. SAN Fibre committee are defining standards to converge the two into
Channel links are being designed and deployed today to a unified fabric with Fibre Channel over Ethernet (FCoE).
support migration to 16G. Maximum 16G OM3 and OM4
channel distances are 100 m and 125 m, respectively. Fibre FCoE is simply a transmission method in which the Fibre
Channel single-mode fiber usage is minimal in the data Channel frame is encapsulated into an Ethernet frame
center but is exclusively used for synchronization between at the server (Figure 2.5). The server encapsulates Fibre
primary and secondary data center sites. T11 activity has Channel frames into Ethernet frames before sending
recently started to develop 32G guidance. Initial objectives them over the LAN and de-encapsulates them when
are for a duplex fiber serial transmission solution with FCoE frames are received. Server I/O consolidation
OM3 and OM4 fibers for 70-100 m distance. Table 2.2 combines the NIC and HBA cards into a single converged
provides the T11 Fibre Channel speed roadmap. network adapter (CNA) which reduces server cabling and
power/cooling needs. At present, the Ethernet frame is
Fibre Channel over Ethernet removed at the Ethernet edge switch to access the Fibre
Data centers utilize multiple networks that present opera- Channel frame which is then transported to the SAN
tional and maintenance issues as each network requires dedi- directors. FCoE encapsulation standards activity takes
cated electronics and cabling infrastructure. As previously place at the Fibre Channel T11.3 committee.
discussed, Ethernet (LAN) and Fibre Channel (SAN) are
the typical networks in a data center. Fibre Channel’s T11
technical committee and the Institute of Electrical and
Ethernet
Channel
Header
Header
Header
FCoE
Fibre
CRC
EOF
FCS
Figure 2.5
Fibre Channel Payload | Drawing ZA-3673
TABLE 2.3: T11 Fibre Channel Speed Roadmap
Figure 2.8
Third Generation FCoE Architecture | Drawing ZA-3471
Chapter Three:
Fiber Type and Performance
As fiber becomes more widely deployed in the data center, include a minimum 2000 MHz•km effective modal band-
a system designer should evaluate all the various grades of width (EMB) for OM3 and 4700 MHz•km EMB for OM4.
multimode fiber optic cable to ensure the data center will The OM multimode fiber nomenclature originated in the
support current and future data rates. As data rates and ISO/IEC-11801, second edition standard and has been
the physical size of data centers increase, the need for adopted into TIA standards such as TIA-568, Rev C.3.
designing a bandwidth and link-length scalable network In addition to OM3 and OM4, OM1 and OM2 designations
is more important then ever. The purpose of this chapter are included for standard 62.5 µm and 50 µm multimode
is to familiarize the reader with OM3 and OM4 fiber types fibers, respectively. See Table 3.1.
and performance requirements needed to support local
area network (LAN) and storage area network (SAN) Data center high data rates in conjunction with the desired
applications commonly used in data centers. application distances support OM3 and OM4 as the default
choice fiber types. The small core size of 50/125 µm fiber
OM3/OM4 Laser-Optimized yields an inherent higher bandwidth capability than other
50/125 µm Multimode Fiber multimode fibers such as OM1 fiber. Tables 3.2 and 3.3
Data center LAN and SAN networks should be designed provide OM3 and OM4 fibers distance capabilities for
to support legacy applications as well as emerging high- Ethernet and Fibre Channel data rates.
data-rate applications. The emergence of high-data-rate
systems such as 10, 40 and 100 Gigabit Ethernet and 8 and Corning Cable Systems strongly recommends OM3 and
16 Gigabit Fibre Channel has resulted in OM3 and OM4 OM4 fibers for the data center. When compared to OM1
multimode fibers being the dominant optical fiber types and OM2 multimode fibers, OM3/OM4 fibers have
deployed in the data center. the highest 850 nm bandwidth to accommodate longer
distances, provide more system budget margin and support
The TIA-492AAAC OM3 detailed fiber standard was migration to higher data rates such as 16/40/100G.
released in March 2002, and the TIA-492AAAD OM4
detailed fiber standard was released in August 2009. The
fibers are optimized for laser-based 850 nm operation and
TABLE 3.1
Overfilled Modal Effective Modal
Optical Fiber
Fiber Reference Wavelength Bandwidth-Length Bandwidth-Length
Cable Type
Product (MHz•km) Product (MHz•km)
TIA-492AAAA-A
62.5/125 µm 850 200 Not Required
IEC 60793-2-10
multimode (OM1) 1300 500 Not Required
Type A1b
TIA-492AAAB
50/125 µm 850 500 Not Required
IEC 60793-2-10
multimode (OM2) 1300 500 Not Required
Type A1a.1
850 µm TIA-492AAAC-A
850 1500 2000
laser-optimized IEC 60793-2-10
1300 500 Not Required
50/125 µm (OM3) Type A1a.2
850 µm TIA-492AAAD
850 3500 4700
laser-optimized IEC 60792-2-10
1300 500 Not Required
50/125 µm (OM4) Type A1a.3
TABLE 3.2: 850 nm Ethernet Distance (m)
1G 10G 40G 100G
Expectation is that implementing an OM3/OM4 physical outside sources, fiber produces no electronic emissions,
layer solution should provide a 10-15 year service life therefore it is not a concern of the Federal Communications
without recabling. Commission (FCC) or European emissions regulations.
Cross-talk does not occur in fiber systems and there are no
Cable, connectors, hardware and electronics are now readily shared sheath issues as with multipair unshielded twisted-pair
available to support usage of these 50 µm fibers. The techni- (UTP) copper cables. Also, standards activity has shown
cal and commercial community has recognized the benefits evidence of alien cross-talk between UTP copper cables that
of OM3/OM4 as the fibers have been adopted into IEEE cannot be corrected by electronic digital signal processing
40/100G and Fibre Channel 4/8/16G transmission stan- (DSP). Because all-dielectric cables, as well as the new
dards as well as the TIA-568-G3 structured cabling and dielectric armored cables, can be used, grounding concerns
connectivity standards. The 850 nm wavelength now offers can be eliminated and lightning effects dramatically reduced.
and will continue to offer the most economical solution Optical fibers are virtually impossible to tap, making it the
for data center applications based on electronic costs. most secure media type. Most importantly, optical bandwidth
The data rate scalability of OM3 and OM4 fibers provides cannot be adversely affected by installation conditions.
the ultimate media solution for data center managers to Compare this to the copper system impairments that an
ensure their structured wiring systems support legacy as installer can impact.
well as future application needs.
10G Electronics and Cooling –
Fiber vs. Copper The Optical Advantage
A well-planned structured cabling system in the data center 10G optical switch electronics and server adapter cards
will support both the applications of today as well as the require less power to operate compared to 10G UTP cop-
future. Corning Cable Systems’ data center solutions do per. The high insertion loss of copper cables at the extended
just that, allowing today’s systems to grow gracefully as frequency range needed to support 10G and the required
requirements change without concern of obsolescence. electronic digital signal processing (DSP) noise-reduction
Fiber is the most attractive medium for structured cabling circuitry means that energy consumption will inevitably be
because of its ability to support the widest range of applica- higher than that of low-loss fiber interconnects. 10GBASE-
tions at the fastest speeds for the longest distances. SR SFP+ optical transceivers consume a maximum of 1.0
Additionally, fiber has a number of intrinsic advantages watt (typical 0.5 watt) per port compared to 6-8 watts per
beneficial to any application at any speed. Fiber is immune port for a 10GBASE-T copper switch. SFP+ chassis line
to electromagnetic interference (EMI) and radio frequency cards are intended to support up to 48-64 ports, while
interference (RFI), therefore its signals cannot be corrupted 10GBASE-T cards are expected to have 8-16 ports.
by external interference. Just as it is immune to EMI from 10GBASE-SR server adapter cards typically use less
than nine watts to service up to 300 m, while announced
10GBASE-T cards use 24 watts to service up to 100 m. As network speed grows, optical fiber
Experts have stated that 10GBASE-T over CAT 6A or offers significant advantages over copper
CAT 7 twisted-pair can extend up to 100 m, but power
requirements hinder its cost-effectiveness. A 10G optical 10 Gbps Example
system requires far fewer switches and line cards for equiva- 90%
lent bandwidth capability of a 10G copper system. Fewer
240
288
144
192
High fiber density, combined with the small diameter of
96
48
optical cable, maximizes raised floor pathways and space
utilization for routing and cooling. Optical cables also offer Number of 10G Ports
superior pathway usage when routed in aerial cable trays. Figure 3.1
A 0.7-inch diameter optical cable would contain 216 fibers Electronics and Cooling Savings
to support 108 10G optical circuits. The 108 copper cables
required to provide equivalent capability would have a
5-inch bundle diameter. The 10G twisted-pair copper cooling damming effects and interference with the ability
cable’s physical design contributes to major patch panel of ventilation systems to remove dust and dirt. Optical
and electronic cable management problems. The larger cable offers significantly better system density and cable
CAT 6A outer diameter impacts conduit size and fill ratio management and minimizes airflow obstructions in
as well as cable management due to the physical size and the rack and cabinet for better cooling efficiencies.
increased bend-radius. Copper cable congestion in pathways See Figures 3.2 and 3.3.
increases the potential for damage to electronics due to air
Figure 3.4
Relative Cost of Single-Mode vs. Multimode 10G Transceiver
OM3/OM4 EMBc To ensure field performance, EMB is calculated for 10
For systems operating at data rates greater than 1 Gb/s, actual laser sources which have been determined to repre-
TIA/EIA-455-220 and IEC 60793-1-49 bandwidth test sent the performance extremes of all encircled compliant
methods are used to measure the fiber effective modal VCSELs. Of these 10 sources, the one yielding the lowest
bandwidth (EMB) that include a series of small spot size EMBc value is taken to represent the minimum expected
launches (approximately 5 µm) indexed across the fiber core. performance level of all standards-compliant VCSELs, and
Measurements are made of the output pulse time delay and the EMBc value associated with this source is therefore
mode coupling power of the fiber as a function of radial referred to as the minimum calculated EMB or minEMBc.
position. These measurements are referred to as differential
mode delay (DMD) measurements. Data from these meas- The primary advantage of the minEMBc method over the
urements can be analyzed by two methods to determine DMD mask method is that the minEMBc method guaran-
whether the fiber meets the EMB requirement of a specific tees standards-compliant fiber performance under worst
application. The first method for translating DMD meas- case source/fiber interactions while providing an actual
urements into an EMB prediction is commonly referred to value of bandwidth in the scalable units of MHz•km. The
as the DMD mask approach, where the leading and trailing minEMBc value can then be used to calculate bit rates and
edges of each pulse are recorded and normalized in power link lengths for systems requiring EMB values other than a
relative to each other. This normalization approach reduces minimum 2000 MHz•km. Corning Cable Systems recom-
mends that multimode fiber intended for current or future
use at data rates ≥ 1 Gb/s should be specified according to
the raw DMD data to focus exclusively on time delay, where
the overall fiber delay is calculated as the difference between
the times for the slowest trailing edge and the fastest leading minEMBc values rather than pass/fail performance indicated
edge in units of ps/m. In order for a fiber to be determined by the DMD mask method.
as meeting the required minimum value of 2000 MHz•km
EMB for OM3 at 850 nm, the DMD data must conform to
one of six templates or masks and must not show a DMD
measurement greater than 0.25 ps/m for any of four speci-
fied radial offset intervals. In order for a fiber to be deter-
mined as meeting the required minimum value of 4700
MHz•km EMB for OM4 at 850 nm, the DMD data must
conform to one of three templates or masks and must not
show a DMD measurement greater than 0.11 ps/m for any
of four specified radial offset intervals. It should be noted
that this method provides only a pass/fail estimation against
the 2000 MHz•km and 4700 MHz•km requirements.