White Paper

100 Gigabit Ethernet – Fundamentals, Trends and

Measurement Requirements
By: Peter Winterling

Nearly every two years, a new hierarchy level is announced for telecommunications. The introduc-
tion of the 40 Gbps technology has dominated telecommunications for the last two years and now
everyone is talking about 100 Gbps as the next generation. On the surface, everything looks simply
like a change of generations is taking place and 40 Gbps seem to be outdated already. However,
with the introduction of the 100 Gbps technology, things are quite different compared to the intro-
duction of previous generations.
If we look at this more closely, the terms 40 Gbps and 100 Gbps are general terms for a comprehensive
technological change in transmission technology. Particularly during this phase of implementing new
technology, the measurement technology plays an extraordinarily important role.
The transition from Gigabit Ethernet to 10 Gigabit Ethernet brought the Ethernet technology to the transport
networks and, from the aspect of the transmission protocol, this represented a more important milestone than
the introduction of the 100 Gigabit Ethernet technology. The standardization is still not finalized; however, no
significant changes in relation to the 10 Gigabit Ethernet technologies are expected. Anything revolutionary
will be determined by the physical parameters. It already became clear with the 40 Gbps technology that
transmissions in existing transmission infrastructure are possible only with substantial modifications to the
optically transmitted signal and path. Now, all possibilities must be exhausted for the transmission of 100-Gigabit
Ethernet, or rather OTU4, in wide-area networks.

Standardization at 100 Gigabit Ethernet

Three organizations are involved in the standardization for 100 Gigabit Ethernet: IEEE’s Higher Speed Study
Group (HSSG) defines the Ethernet specifications under the term 802.3ba. At ITU-T, the SG15 standardization
group deals with the integration of the 100 Gigabit and 40 Gigabit Ethernet signals within the OTN framework.
At OIF (Optical Internetworking Forum), the PLL (Physical and Link Layer) work group is working on the
integration of these signals in the DWDM technology of the metro and wide-area network technology.
As initial position for the standardization of 100 Gigabit Ethernet, it was specified to retain the past frame sizes
and frame formats of the IEEE 802.3 standard. For the MAC layer, the target is a transmission quality with a bit
error rate of less than BER = 10- 12. Efforts will be made to use the OTN technology as a transport medium and to
support it with corresponding specifications.
100 Gigabit Ethernet is transmitted for the time being via several optical channels using the multi-lane concept.
Three different physical interfaces are standardized, see table. Two ranges are defined for the transmission via a
single-mode fiber:

WEBSITE: www.jdsu.com/test
 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 2

Physical Medium Device (PMD) 40 Gigabit Ethernet 100 Gigabit Ethernet

Multimode optical fiber < 100 m 40GBase-SR4: 100GBase-SR10:
with OM3 fiber • Ribbon fibers • Ribbon fibers
• 850 nm • 850 nm
• 4 × 10 Gbps • 10 × 10 Gbps
Single-mode optical fiber > 10 km 40 GBase-LR4: 100GBase-LR4:
• CWDM 20 nm (channel spacing) • LAN-WDM 4.5 nm
• 1310 nm • 1310 nm
• 4 wave lengths × 10 Gbps • 4 wavelengths
• 25 Gbps
Single-mode optical fiber > 40 km 100GBase-ER4:
• LAN WDM 4.5 nm
• 1310 nm
• 4 wave lengths × 25 Gbps
Single mode optical fiber, Not standardized:
not standardized • Channel spacing 8 nm
• 1550 nm
• 10 Wave lengths × 10 Gbps
Table. Standardization for 40 Gigabit Ethernet and 100 Gigabit Ethernet

Figure 1 Optical Network tester ONT-503 with 100-Gigabit-Ethernet-Module and CFP

100GBase-LR4 (long range) describes the optical interface for four wavelengths within the range of 1310 nm
with a channel spacing of 4.5 nm with an attainable transmission range of 10 km and 100GBase-ER4 (extended
range) for a range of 40 km. For the first tests, yet another optical interface with ten wavelengths in the range
of 1550 nm is available. This variant, which was not standardized, however, is technically the simpler solution
since the multiplexers (“Gearbox”) can be omitted in these transponders. For short connections within a
computer center, another interface for multimode fibers is defined in the wavelength range of 850 nm. Here, ten
transmission channels run parallel in a cable by means of ribbon fibers. The transmission channels are plugged
together in an optical plug. By using OM-3-fibers, transmission lengths of at least 100 m can supposedly be
achieved.
In addition to the optical interfaces, an electrical interface was also defined for both data transmission rates.
Thus, a transmission range of maximum 10 m should be able to be obtained by four or ten parallel signals.
The optical interfaces for transmitters and receivers are integrated in so-called CFP modules (100 Gigabit small
form factor pluggable). Similar to the pluggable XFP modules (10 Gigabit small form factor pluggable), they are
 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 3

also transponders that can be changed during operation (“hot swappable”) or that are independent of protocol.
The four DFB lasers with their corresponding higher optical power output require efficient heat management
for the CFP module. It is plugged into a precision shaft, its connection ensuring corresponding temperature
heat dissipation. Fig. 1 shows the JDSU ONT-503 optical network tester with the 100 Gigabit Ethernet plug-
in module. The CFP module is clearly visible. Fig. 2 shows the OSI reference model as per IEEE 802.3ba for
100 Gigabit Ethernet.
OSI reference model IEEE 802.3 reference model

Application Upper protocol layers

Presentation MAC client
Session MAC
100 Gbps
Transport Reconciliation
Network CGMII
Data link 100GbE PCS (66B/64B)
Physical
CAUI

PMA Common electrical I/F

10 x 10.3 Gbps pairs
PMD

MDI
Different options for
n Lanes, 100GbE transmission
n = 4 or 10

Fig. 2 OSI reference model and Ethernet PHY reference model as per IEEE 802.3ba

Besides the optical transmitters and receivers, the CFP modules additionally contain the multiplexer/
demultiplexer, also called the “Gearbox”. Thus, the 20 virtual data streams which are structured in 10 physically
parallel data streams of 10 Gbps each, are combined by the MAC layer into four physical data streams of 25 Gbps
each. Multiplexers and demultiplexers require space in the CFP module and contribute significantly to the
temperature budget. Long-term, these functions will migrate to the ASIC of signal processing. This requires an
appropriate board layout and a connecting path for transferring to the optical interfaces for 25 Gbps or 28 Gbps
in place of 10 Gbps. Therefore the optical interfaces can turn out significantly smaller than today’s CFP modules
and the overall design becomes significantly less expensive. Possible names for it are CxFP or QSFP.
The interfaces and the management functions of this module are described in detail in the CFP MSA
Management Interface Specification [1].
Client application

MAC Other clients

CGMII Adaptation Adaptation

20 virtual lanes 20 logical lanes

PCS Adaptation OPU4

G.709 Appendix VII: ODU4
20 virtual lanes
Adaptation of parallel
of 64B/66B
64B/66B encoded clients
block encoded OTU4
data over n
parallel physical
lanes
Striping G.709 Annex C
adaptation of OTU3 and
20 logical
OTU4 over multichannel
lanes
parallel interfaces
Gearbox
10 x 10 Gbps 10 x 11.2 Gbps
CAUI CAUI
PMD PMD Serial TRX

4 or 10 physical IrDI Long-haul

lanes parallel I/F serial

Fig. 3 100-Gigabit-Ethernet and OTN integration

 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 4

The CFP MSA defines a transceiver for 40 Gbps as well as for 100 Gbps, including its use for High-Speed
Ethernet (40 Gigabit and 100 Gigabit Ethernet). All efforts are directed towards the common agreement to
develop a reasonably priced 100-Gbps technology. Optical transponders significantly affect the cost structure of
the transmission system and thus the focus is on them.
For transmission in metro and wide-area networks, the 100 Gbps signal must be integrated in the OTN
according to the ITU-T recommendation G.709. For 100 Gigabit Ethernet, an OTU4 framework is defined and
the description of the physical interface for a transmission in four parallel channels was expanded accordingly
[2]. The different variants are shown in Fig. 3. The possibility of integrating other client signals in OTU4 is
enabled explicitly.

Fig. 4 Multi-lane distribution (MLD) tests for 40/100 Gigabit Ethernet

With this first step of parallel transmission, an essential goal of increasing the spectral efficiency in wide-area
networks has not yet been attained. Today, DWDM networks are primarily realized with a channel spacing of
50 GHz. Today, with adequately high-level modulation procedures (e.g. DQPSK) a 40 Gbps signal can already
be transferred in a channel of the OTU3 framework [3]. With these admittedly very complex and accordingly
expensive transponders, the transmission systems can bridge distances of more than 100 km. For quite some
time, a well-known large network operator has been equipping his wide-area network with this technology.
With 100 Gbps becoming possible, there are two goals: on the one hand, the technical development of the
transmission systems should make much more economical transponders available. Later, these transponders
will enable serial transmission in a channel by means of higher modulation methods. Only then will it be possible
to obtain a higher spectral efficiency. The CFP MSA explicitly describes the 40 Gigabit Ethernet as an interface
as well, which is likewise described in the 802.3ba IEEE standard, Fig. 1. On the other hand, significantly cheaper
transponders are required for the 40 Gbps level, because 100 Gigabit Ethernet does not replace this technology
by any means. The balancing act of ever smaller optical transponders, which are still required to cope with large
ranges of wide-area networks having a high spectral efficiency, is not yet resolved and technical experts will
ponder this problem for some time to come.

Test Requirements for CFP Modules for 100 Gbps

More than ever, the introduction of 100 Gigabit Ethernet requires a measurement technology that has been
customized appropriately in order to accompany the stage-by-stage introduction.
Manufacturers of components and systems first of all require measurement engineering for 100 Gbps. For testing
the CFP modules used in the transmission systems, a 100 Gigabit Ethernet signal must be produced with ten
parallel electrical connections that are coded as 20 virtual channels. The Ethernet signal coming from the MAC
layer is not firmly allocated to the virtual channels. In accordance with the specification, the virtual channels on
the transmitting end can be shifted as desired at the entry of the multiplexer. They are being sorted as per the
so-called round-robin principle. The receiver must synchronize itself automatically. It must be possible to set any
 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 5

configuration at the time of the test. Fig. 4 shows the adjustment possibilities on the transmitting end.
At the optical interfaces of the CFP, the analyzer verifies the multiplex function and correspondingly tests the bit
error rate of the client signal. The value of BER = 10–12 must be adhered to, which is possible only by the use of
forward error correction (FEC). This can be checked by displaying the bit errors in the PCS layer. The analyzer
must of course make the optical parameters of its own CFP available over the MDIO management interface, as
the information about the exact wavelength and the optical output levels should be known.
For IEEE-compliant CFPs, it is extremely important to test the skewing for each individual optical channel as
well as the error-free acceptance on the receiver side. This is statically possible by inputting the number of bits
to be moved in the individual physical channels; as per the 802.3ba IEEE recommendation, dynamic skewing is
described in addition, which requires an external timing distribution method.
All these tests, which are required for 100 Gigabit Ethernet for the first time because of the parallel transmission,
are an integral part of testing the function of CFP modules. Using one of the first CFPs commercially available
from a reputable manufacturer, JDSU has shown its compliance with standards at the end of 2009.

Stressed Receiver Sensitivity for Transponders

For SDH and OTN, the standardization for jitter measurements is given in the O.172 ITU-T recommendation.
However, this cannot easily be adopted for the package-oriented Ethernet transmission. Therefore, the IEEE
described a “Stressed Receiver Conformance Test” in the 802.3ae standard (chapter 52): A reference signal
is changed with different parameters both in amplitude as well as in frequency, with the goal of simulating the
transmission characteristics through a glass fiber route as realistically as possible. The bit error rate is measured
as a function of these parameters. Additionally, the eye diagram could be analyzed with a fast digital oscilloscope.
For years, JDSU’s Hydra testing system has been used to determine the stressed receiver sensitivity of 10 Gbps
SFP modules for characterizing transponders. Fig. 5 shows a solution for 100 Gigabit Ethernet using the multi-
lane concept. Optical modules contained in the MAP-200 measuring platform take over demultiplexing and
multiplexing and select the parallel wavelength to be affected. In the transition mode of the Hydra testing system,
the signal of a wavelength is controlled in accordance with the requirements of SRS testing. The ONT-503
network tester will send a 100 Gbps signal to the CFP module via the CAUI interface and determine the bit errors
by means of the returned signal

Stress generator
D Optical O/E Stress selection:
E switch – LR, ER, Off
4 λ x 25G M
U 1x4 Stress types:
X – Jitter
– VECP (BT filter)
– Opt. channel power
– Dynamic skew
indent (<15 UI)
attenuation

M
Optical

4 λ x 25G
U
1 stressed X
E/O

Hydra

MAP-200

Fig. 5 “Stressed eye testing” in case of four wavelengths with respectively 25 Gbps

Although the IEEE does not have a final standardization for the stressed receiver sensitivity measurement, JDSU
has already presented a solution with a first approach for testing.
 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 6

Line Cards and Interaction in the Transmission System

Along with the described transponder tests, the complete test of network elements for 100 Gigabit Ethernet also
includes the validation of the PCS layers and an Ethernet IP protocol test that is as comprehensive as possible.
For this purpose, four (for 40 Gigabit Ethernet) or ten (for 100 Gigabit Ethernet) parallel interfaces must
currently be adapted and analyzed in the MAC/IP layer. The configuration options and their evaluation do
not differ from the analysis of 10 Gigabit Ethernet. For example, it should therefore be possible to produce and
analyze 256 independent streams with different traffic profiles. VLAN tagging, “MAC in MAC” and GMPLS
analysis are just as necessary as statistical evaluations regarding frame losses, latency, package jitters and service
disruption, Fig. 6. Basic tests, for example, are an extensive quality of service (QoS) analysis, along with special
test frameworks and a corresponding bandwidth evaluation (bandwidth utilization).
Overview of all streams

Packet jitter
per stream

Delay time
per stream

Statistics per stream

Fig. 6 Complete Ethernet test depth with the tester ONT-503

The next step, the integration of the Ethernet signal within the OTU4 OTN- framework for the transmission
in metro and wide-area networks, Fig. 4, first uses four parallel transmission paths. The nomenclature for
such regenerated signals corresponds to OTL k. n (Optical Channel Transport Lane), wherein k stands for the
hierarchy level and λ for the number of wavelengths at the optical interface. For OTU4 with a bit rate of 112 Gbps
and four wavelengths, the physical interface would then be OTL4.4; for OTU3 with 43 Gbps, it would be OTL3.4.
Fig. 7 shows the variants, which are tested accordingly.
Just like in low hierarchies, the wrapper or de-wrapper function is an important test in the OTN layer as
well. The test examines whether or not the client signal is correctly mapped within the OTN frame and if the
bits for the forward error correction were produced and if the output signal complies with the G.709 ITU-T
recommendation and vice versa.
Long range
Client modules
(proprietary)
CAUI
OTU4
4 x 25 Gbps
OTU4
…. Line
CAUI framer
module Short range
10 x 10 Gbps FEC

112 Gbps
IrDI
100GbE Frames

OTU4 frames

JDSU ONT-503
100G transmission tester

Fig. 7 100 Gigabit Ethernet and OTN interworking test

 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 7

The test becomes significantly more complex if the client signal in one of the next steps is no longer exclusively
an Ethernet signal. By means of the current ITU-T standardization, even ODU multiplexing is possible using
lower hierarchies, e.g. OTU3. In the OTN layer, the tester must map all the changes for G.709 as well. An extreme
example is the concentration of 80 different ODU0 frames with one Gigabit Ethernet signal each as a payload in
an OTU4 frame [4]. Fig. 8 shows the test variants and interfaces in a Netscape.

The Multi-Lane-Concept in the Metro and Wide Area Network

So far in a DWDM system, a transmission channel could be transferred with each wavelength using the fixed
channel spacing. A 40 Gbps signal can only be transmitted into the conventional 50 GHz channel spacing using
high-level optical-modulation procedures [4]. This will not be different for the 100 Gbps technology; instead,
more complex modulation procedures, e.g. the polarization multiplex, will be used. However, it is new that four
wavelengths are required for the first step of the implementation of the 100 Gigabit Ethernet for distances of up to
10 km (in metro networks, therefore).
The Optical Internetworking Forum (OIF) is working on the integration of this multi-wavelength transmission
system into the existing infrastructure. There are questions to be answered, such as how channel groups are to be
organized in existing DWDM systems, how an equivalent network works, how this multi-lane concept is to be
integrated in a network management system, etc. Apart from these structural questions, there are also additional
physical hurdles, which are developing during the transmission. The Polarization Mode Dispersion (PMD)
and the Chromatic Dispersion (CD) are fiber characteristics, which can severely limit the transmission range of
signals with high bit rate.
Conventional systems are already compensating for the static chromatic dispersion and the introduction of
multi-lane systems raises the question about the acceptable tolerance for the compensation of all wavelengths of
the DWDM system.
Router Router Switch
Router <=> Router via 100 GE Router <=> switch via 40GE

CFP 4x10 G CFP

4 x 25 G 4 x 10 G
CxFP/QSFP

CFP
4 x 28 G Router <=> DWDM
via Inter-domain OTU4
(or 100GbE)

100 G line
modulation
and FEC
10GbE, SDH, OTN

DWDM equipment

Fig. 8 Interworking tests at 100 Gigabit Ethernet and OTN layer

In contrast, the PMD is changeable and is dependent on external influences such as changes of temperature and
the mechanical load of the fiber. For example, the vibration in guard wires of high-voltage transmission lines of
integrated fibers or cables over bridges and in underground shafts. Since the PMD varies differently not only over
time but also with different wavelengths, special attention must be given to it in case of multi-lane systems.
The average value of the PMD as a function of the wavelength is determined by using the measurement
procedure common today for fiber characterization after the installation of the cable. In the case of the single-
channel system, the maximum limiting value of the dispersion is calculated by the degradation of the quality of
service (QoS) of 1 dB (10 ps for 10 Gbps systems, for example). Additional criteria must be found for multi-lane
systems. The use of coherent optical receivers with subsequent digital signal processing considerably diffuses the
dispersion issue. It is questionable whether this still-expensive technology is already being used in multi-lane
concepts or if it will be reserved for the future serial transmission of OTU4 with only one wavelength.
 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 8

100 Gigabit Ethernet Still Has a Long Way to Go

In contrast to all other earlier generation steps, the 100 Gbps technology is being realized for the first time for
short transmission paths and first will be applied in computer centers for networking powerful computers.
Interfaces defined for ranges of 10 km and 40 km will first be geared for use in the metro network. But whether
in computer centers or in the metro network, the 100 Gbps technology will be introduced with parallel
transmission and for that to happen, a workable concept is required for the integration of a multi-lane system
into the existing optical DWDM networks. In the next generation, it will be possible to use 100 Gbps for serial
transmissions. However, many development steps are necessary in order to achieve these goals: the cost-effective
implementation of higher-level optical-modulation procedures (including polarization multiplexing) and the
realization of fast signal processors to smooth the way for coherent receivers.

Fig. 9 Optical network tester ONT-503 even supports the 100 Gigabit Ethernet

For all these steps, a measurement technique will be necessary, which will accompany this development. It will
gain insights into the measuring technology required later to measure the operation. The basic equipment of the
ONT-503 tester with its different modules makes available almost all interfaces for the SDH and OTN hierarchy,
from STM-1 to OTU4 and now also 10 Gigabit Ethernet and 100 Gigabit Ethernet, Fig. 9. JDSU supports the
future telecommunications technologies on all levels – with the development of optical components for the
systems of tomorrow as well as with measuring techniques in order to test the transmission technology and to
enable its integration into the existing networks.
 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 9

Abbreviations Abbreviations
ASIC Application Specific Integrated Circuit SFP Small Form-Factor Pluggable
BER Bit Error Ratio SG Study Group
CAUI 100 Gigabit Attachment Unit Interface STM Synchronous Transport Module
CD Chromatic Dispersion TK Telecommunication
CFP 100 Gigabit Small Form Factor VC Virtual Container
Pluggable VCAT Virtual Concatenation
CGMII 100 Gigabit Media Independent VLAN Virtual Local Area Network
Interface XFP 10 Gigabit Small Form Factor Pluggable
DFB Distributed Feedback (Laser)
DWDM Dense Wavelength Division
Multiplexing
DQPSK Differential Quaternary Phase Shift
Keying
ER Extended Range
FEC Forward Error Correction
GbE Gigabit Ethernet
GFP Generic Frame Procedure
GMPLS Generalized Multi Protocol Label
Switching
HSSG Higher Speed Study Group of IEEE
IEEE Institute of Electrical and Electronics
Engineers
IrDI Inter Domain Interface
ITU-T International Telecommunication Union
–Telecommunication Standardization
Sector
LAN Local Area Network
LR Long Range
MAC Medium Access Control
MDI Media Dependent Interface
MDIO Management Data Input/Output
MII Media Independent Interface
MLD Multi Lane Distribution
MSA Multi-Source Agreement
ODU Optical Data Unit
OIF Optical Internetworking Forum
OM3 Optical Multimode Level 3
OPU Optical Payload Unit
OTL Optical Channel Transport Lane
OTN Optical Transport Network
OTU Optical Transport Unit
PCS Physical Coding Sub layer
PMD Physical Medium Device
PMD Polarization Mode Dispersion
PMA Physical Medium Attachment
PLL Physical and Link Layer
QoS Quality of Service
SDH Synchronous Digital Hierarchy
 White Paper: 100GE Fundamentals, Trends, and Measurement Requirements 10

References
[1] CFP MSA Management Interface Specification. www.cfp-msa.org/Documents/CFP_MSA_Management_
Interface_Specification_Draft_1_2_Public_B.pdf
[2] Winterling, P.: OTN as Transport medium of the future and requirements of the instrumentation. Trade
journal for information and communication technology 62 (2009) H. 6, S. 28 – 32
[3] Seimetz, M.: High-value modulation procedure in the optical glass-fiber transmission technology. Part 1 and
part 2. Trade journal for information and communication technology 62 (2009) H. 6, S. 34 – 36 and H. 7-8, S.
20 – 23
[4] Van den Borne, D. u.a.: Polmux-QPSK modulation and coherent detection: the challenge of long-haul 100G
transmission. Proc. ECOC 2009. Berlin • Offenbach: VDE VERLAG, 2009
[5] Gavioli, G. u. a.: 100Gb/s WDM NRZ-PM QPSK Long-Haul Transmission Experiment over Installed
Fiber Probing Non-Linear Reach With and Without DCUs. Proc. ECOC 2009. Berlin • Offenbach: VDE
VERLAG, 2009
[6] Brooks, P.: Meeting the test challenges of 100-Gigabit Ethernet. Lightwave 26 (2009) H. 12, S. 29 – 31, 35 and
39
[7] ITU-T Document G.Sup43: Transport of IEEE 10GBase-R in optical transport networks (OTN)
[8] Ziemann, O.: When comes the Terabit-Ethernet – or the mathematical gadget of the extrapolation. Trade
journal for information and communication technology 62 (2009) H. 5, S. 26 – 27
[9] Kiefer, R.; Winterling, P.: DWDM, SDH & Co. Technology and Troubleshooting in optical Networks. 2.,
neubearb. u. erw. Edition, 2002, Bonn: VMI Industrial book Abbreviations

Test & Measurement Regional Sales

NORTH AMERICA LATIN AMERICA ASIA PACIFIC EMEA WEBSITE: www.jdsu.com/test

TEL: 1 866 228 3762 TEL: +1 954 688 5660 TEL: +852 2892 0990 TEL: +49 7121 86 2222
FAX: +1 301 353 9216 FAX: +1 954 345 4668 FAX: +852 2892 0770 FAX: +49 7121 86 1222
Product specifications and descriptions in this document subject to change without notice. © 2010 JDS Uniphase Corporation 30168186.000.0910 100GEFUND.WP.LAB.TM.AE September 2010