Carrier Ethernet

Operations, Administration & Maintenance

Santanu Dasgupta

1
Acknowledgement

  Jose Liste
  Sudarshan Pathalam

2
Agenda
  OAM Overview
  Ethernet OAM – Protocol Overview
IEEE 802.3ah – Link OAM, also known as EFM
IEEE 802.3ag – Connectivity Fault Management (CFM)
ITU Y.1731 – Fault & Performance Management
MEF E-LMI – Configuration Management

  Ethernet Service Activation Testing

  OAM Interworking
  Sample Fault Management Scenarios
  Ethernet OAM Deployment Use Cases
  Summary

3
Operations, Administration & Maintenance

  F – Fault Management

FCAPS
  C – Configuration Management
  A – Accounting
  P – Performance Management
  S – Security Management

OAM Protocols and Mechanisms helps operator to

achieve some of the FCAPS functionality
OAM capability is one of the key differentiator to
make a SP network truly “Carrier Grade”
4
 Problem Scope
A few possible scenarios

Excessive Uni-directional D-LDP session

Encoding Errors Link failure Duplex
SP Network mismatch
Access
Excessive Core VC failure
Customer
Equipment FCS Errors U-PE D C-VLAN
N-PE 1 N-PE 3
U-PE A MPLS to EVC
CE
Cross-connected Unexpected SONET/SDH mismatchCE
Native LSP Speed
Service Ethernet Endpoint Failure mismatch
PW
CE N-PE 4 CE
U-PE B MPLS U-PE C
N-PE 2
AC
Physical
Ethernet UNI failure
Ethernet UNI

link failure PW failure

Physical VLAN to VFI
link failure xconnect failure
UNI VFI
port failure failure
Physical
U-PE P-Router link failure
device failure failure
5
 OAM: The Concept
  Operations, Administration, Maintenance & Provisioning:
− fault indication − performance monitoring
− security management − diagnostic functions
− configuration & service provisioning

  OAM covers both Nßà S and WßàE interfaces

Management Plane

N ßà S
(NMS / EMS)

Network Plane
(Elements)

W ßà E
6
Scope of Ethernet OAM
  Operations, Administration, Maintenance & Provisioning:
− fault indication − performance monitoring
− security management − diagnostic functions
− configuration & service provisioning
  OAM covers both Nßà S and WßàE interfaces

Management Plane

N ßà S
(NMS / EMS)

Primary Focus of Ethernet OAM protocols is on W ßà E interactions

(across NEs)

Network Plane
(NEs)

W ßà E

7
 Focus Areas of Ethernet OAM
UNI UNI
Access Aggregation Edge Core Edge Aggregation Access

U-PE PE-AGG N-PE N-PE PE-AGG U-PE

Core Core

EPL   EVPL   EP-­‐LAN   EVP-­‐LAN   EP-­‐TREE   EVP-­‐TREE   AC  EPL   AC  EVPL  

CE Services &
Network Infrastructure

Fault Management Config Management Performance Management

Fault Detection Frame Loss Measurement
Fault Notification Delay Measurement
Service Provisioning
Fault Verification Delay Variation Measurement
Fault Isolation Availability Measurement
Fault Recovery

8
 Ethernet OAM
Building Blocks

Performance Fault
Management Management

Performance
Performance Fault

Management
Management Management

Configuration
LIMI
Configuration
Management

Management
E
LM- A s
E - IP SL Configuration
Management

Y.17 31
.1 a g /
802

a h
802.3
Fault
Performance
Management Management Fault
Management

Configuration
Management

  IEEE 802.1ag: Connectivity Fault Management (CFM)

  IEEE 802.3ah: Ethernet Link OAM (EFM OAM)
  ITU-T Y.1731: OAM functions & mechanisms for Ethernet based networks
  MEF E-LMI: Ethernet Local Management Interface
  IP SLA’s: Performance Management using CFM and Y.1731 mechanisms
9
Carrier Ethernet OAM
Protocol Positioning
Connectivity
Customer Aggregation
Access
E-LMI Core
Fault Management Aggregation Customer
Y.1731 Performance
Business Management Business

MPLS
OAM

Ethernet Ethernet
Ethernet
MPLS/IP MPLS/IP MPLS/IP
Link OAM Residential
Residential

UNI NNI NNI NNI UNI

  E-LMI – User to Network Interface (UNI)

  Link OAM – Any point-point 802.3 link
  CFM, Y.1731 PM – End-to-End UNI to UNI
  MPLS OAM – within MPLS cloud
10
IEEE 802.3ah
(Clause 57)
Link OAM

11
Link OAM (IEEE 802.3ah, Clause 57)
  Provides mechanisms useful
for “monitoring link operation”, OSI Model
such as:
LAN
Link Monitoring Application
CSMA/CD
Remote Failure Indication Layers
Presentation
Remote Loopback Control Higher Layers
  Defines an optional OAM sublayer Session
LLC
  Intended for single point-to-point Transport
OAM (Optional)
IEEE 802.3 links
Network MAC
  Uses “Slow Protocol”1
frames
called OAMPDUs which are Data Link Physical Layer
never forwarded by MAC clients
Physical
  Standardized: IEEE 802.3ah,
clause 57 (now in IEEE
802.3-2005)
(1) No more than 10 frames transmitted in any one-second period
12
IEEE 802.3ah
Key Functions

  OAM Discovery
Discover OAM support and capabilities per device
  Link Monitoring
Basic error definitions for Ethernet so entities can detect failed and
degraded connections
  Fault Signaling
Mechanisms for one entity to signal another that it has detected an error
  Remote MIB Variable Retrieval
Ability to read one or more MIB variables from the remote DTE
  Remote Loopback
Used to troubleshoot networks, allows one station to put the other
station into a state whereby all inbound traffic is immediately reflected
back onto the link
13
IEEE 802.3ah
OAM Events

  Set of events that may impact link operation

  Critical Link events
Link fault—fault in the Rx direction of local DTE
Dying gasp—unrecoverable local failure condition
Critical event—unspecified critical event

  Link events
Errored Symbol Period Event
Errored Frame Event
Errored Frame Period Event
Errored Frame Seconds Summary Event

14
IEEE 802.3ah
Remote Loopback

  Fault localization and intrusive link performance testing

  Loopback Control OAMPDU is used to control a remote
OAM client
  Traffic sent from master loopback port is loopback by
slave port, except Pause and OAMPDU
Slave drops traffic
from LLC on
looped interfaces
Returned traffic
dropped at Master
X to prevent MAC X
OAM OAM
moves
MAC MAC
Slave loops service
PHY frames without PHY
swapping MAC
addresses

Master OAM Slave OAM

15
IEEE 802.1ag: Connectivity Fault
Management (CFM)

16
CFM Overview

  Family of protocols that provides capabilities to

detect, verify, isolate and report end-to-end Ethernet
connectivity faults
  Employs regular Ethernet frames that travel in-band
with the customer traffic
Devices that cannot interpret CFM Messages forward
them as normal data frames

  CFM frames are distinguishable by Ether-Type

(0x8902) and dMAC address (for multicast messages)
  Standardized by IEEE in 2007
IEEE std. 802.1ag-2007

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 17

CFM Overview (Cont.)
Key CFM Mechanisms

  Nested Maintenance Domains (MDs) that break up

the responsibilities for network administration of a
given end-to-end service
  Maintenance Associations (MAs) that monitor service
instances under a given MD
  Maintenance Points (MPs) that generate and respond
to CFM PDUs
  Protocols (Continuity Check, Loopback and Linktrace)
used for Fault Management activities

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 18

CFM Concepts
Maintenance Domain (MD)
CE CE

Operator A Operator B

Service Provider

Customer

  Defined by Operational/Contractual Boundaries

e.g. Customer / Service Provider / Operator

  MD may nest and touch, but never intersect

  Up to eight levels of “nesting”: MD Level (0..7)
The higher the level, the broader its reach

  MD Name Format: null, MAC address, DNS or string-based

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 19
CFM Concepts
Maintenance Association (MA)
CE Operator A Operator B CE

  Monitors connectivity of a particular service instance in a given MD

(e.g. one service traversing four MDs = four MAs)
  Defined by a set of Maintenance End Points (MEP) at the edge of a
domain
  Identified by MAID == “Short MA” Name + MD Name
  Short MA Name Format: Vlan-ID, VPN-ID, integer or string-based
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 20
CFM Concepts
Maintenance Point (MP)—MEP
CE Operator A Operator B CE

MEP MEP

MEP MEP

MEP MEP MEP MEP

  Maintenance Association End Point (MEP)

  Define the boundaries of a MD
  Support the detection of connectivity failures between any pair of MEPs in
an MA
  Associated per MA and identified by a MEPID (1-8191)
  Can initiate and respond to CFM PDUs
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 21
CFM Concepts
Maintenance Point (MP)—MIP
CE Operator A Operator B CE

MEP MIP MIP MEP

MEP MIP MIP MEP

MEP MIP MIP MEP MEP MIP MIP MIP MIP MEP

MIP MIP MIP MIP

  Maintenance Domain Intermediate Point (MIP)

  Support the discovery of paths among MEPs and location of faults along
those paths
  Can be associated per MD and VLAN/EVC (manually
or automatically created)
  Can add, check and respond to received CFM PDUs
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 22
CFM Concepts
UP MEP

  CFM PDUs generated by the

MEP are sent towards the
Bridge’s Relay Function and
not via the wire connected Bridge
to the port where the MEP
is configured
  CFM PDUs to be responded by Relay
the MEP are expected to arrive Entity
via the Bridge’s Relay Function
  Applicable to switches Port A Port B

CFM PDUs

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 23

CFM Concepts
DOWN MEP

  CFM PDUs generated by the

MEP are sent via the wire
connected to the port where
the MEP is configured Bridge / Router
  CFM PDUs to be responded
by the MEP are expected to
arrive via the wire connected Relay
to the port where the MEP Entity
is configured
  Port MEP—special Down Port A Port B
MEP at level zero (0) used
to detect faults at the link
level (rather than service) CFM PDUs
  Applicable to routers
and switches
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 24
CFM Concepts
MAs and UP/DOWN MEPs

DOWN MEP to UP MEP

Applicability of UP/DOWN
MEPs in switches: Bridge 1 Bridge 2
Bridge Bridge Bridge Bridge
  DOWN MEPs are typically used
for MAs spanning a single link Port
Relay
Port Port
Relay
Port
Entity Entity
  UP MEPs are commonly used for
MAs with a wider reach (e.g. end- Monitored area
to-end, beyond a single link)
UP MEP to UP MEP

Bridge 1 Bridge 2
Bridge Bridge Bridge Bridge

Port Port Port Port

Relay Relay
Entity Entity

Monitored area
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 25
CFM Protocols

There are three (3) protocols defined by CFM

  Continuity Check Protocol
Fault Detection
Fault Notification
Fault Recovery

  Loopback Protocol
Fault Verification

  Linktrace Protocol
Path Discovery and Fault Isolation

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 26

CFM Protocols
Continuity Check Protocol
CE Operator A Operator B CE

Catalogue and
Catalogue Catalogue Terminate

MEP MIP MIP MEP

1 2 3
Continuity Check Message
X
(CCM)

  Used for Fault Detection, Notification and Recovery

  Per-Maintenance Association multicast “heart-beat” messages
Transmitted at a configurable periodic interval by MEPs
(3.3ms, 10ms, 100ms, 1s, 10s, 1min, 10min)
Uni-directional (no response required)
Carries status of port on which MEP is configured
  Catalogued by MIPs at the same MD-Level, terminated by remote MEPs
in the same MA
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 27
 CFM Protocols
Loopback Protocol
CE Operator A Operator B CE

S D
MEP MIP MIP MEP

1 1 1. Loopback Message
(LBM)
2 2 2. Loopback Reply (LBR)

  Used for Fault Verification—Ethernet Ping

  MEP can transmit a unicast LBM to a MEP or MIP in the same MA
  MEP can also transmit a multicast LBM (defined by ITU-T Y.1731),
where only MEPs in the same MA respond
  Receiving MP responds by transforming the LBM into a unicast
LBR sent back to the originating MEP
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 28
 CFM Protocols
Linktrace Protocol
CE Operator A Operator B CE

S 6
4 D
2
MEP MIP MIP MEP

1 3 5 1, 3, 5
X Linktrace Message (LTM)
Y 2, 4, 6
Linktrace Reply (LTR)

  Used for Path Discovery and Fault Isolation—Ethernet Traceroute

  MEP can transmit a multicast message (LTM) in order to discover
the MPs and path to a MIP or MEP in the same MA
  Each MIP along the path and the terminating MP return a unicast
LTR to originating MEP
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 29
ITU-T Y.1731 Overview

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 30

ITU-T Y.1731 Overview

  ITU-T recommendation that provides mechanisms for

user-plane OAM functionality in Ethernet networks. It
covers: Ethernet
Service OAM

Fault Management mechanisms MPLS

Service OAM
MPLS-TP IP / MPLS
Performance Management mechanisms Tunnel OAM Tunnel OAM

  Standardized by ITU-T SG 13 in May 2006

Latest published version dated Feb. 2008 after IEEE 802.1ag
standardization

  Frame formats (Multicast Address, Ethertype, and

common OAM PDU fields) and base functionality are
agreed upon across IEEE 802.1ag and Y.1731

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 31

 ITU-T Y.1731 Overview
  OAM Functions for Fault Management
Ethernet Continuity Check (ETH-CC) (Y.1731 adds unicast CCM)
Covered by
IEEE 802.1ag Ethernet Loopback (ETH-LB) (Y.1731 adds multicast LBM)
Ethernet Linktrace (ETH-LT)
Ethernet Remote Defect Indication (ETH-RDI)
Ethernet Alarm Indication Signal (ETH-AIS)
Ethernet Locked Signal (ETH-LCK)
In addition: ETH-TEST, ETH-APS, ETH-MCC, ETH-EXP, ETH-VSP

  OAM Functions for Performance Management

Frame Delay Measurement (ETH-DM)
Frame Loss Measurement (ETH-LM)
Frame Loss Measurement (ETH-SLM)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 32
 ETH-AIS – Alarm Indication Signal
LOC Alarm Suppression

  Without AIS, a MEP would report Loss of Continuity (LOC) for each of its
remote MEPs upon timer expiration
  With AIS, if a MEP receives AIS from the network, it suppresses LOC
alarms from peer MEPs
LOC 1
LOC 3 Network outage
LOC 4 2 AIS defect 2

LOC 1
LOC 2 AIS
mpid = 1 LOC 4 mpid = 1 AIS defect

LOC 2
X 3
AIS
X AIS

3
LOC 3 AIS
LOC 4 AIS defect

4 LOC 1 4
LOC 2
LOC 3 AIS defect

WITHOUT AIS WITH AIS

Timer-driven Event-driven
LOC notification AIS notification
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 33
 Alarm Indication Signal
Transport Path Failure propagation

NMS
AIS Defect Link
Northbound Failure
Syslogs
Customer Operator A Operator B Customer
Equipment Bridges Bridges Equipment

MEP(7)
MIP(7)
MEP(5)
MIP(5)
MEP(3) MEP(2)
MIP(3) MIP(2)
SMEP/link-status

AIS

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 34

Y.1731 – Performance Management Functions

  Frame Loss Ratio – percentage (%) of service frames

not delivered / Total number of service frames delivered
in T time interval
  Frame Delay – round-trip/one-way delay for a service
frame
  Frame Delay Variation – Variation in frame delay
between a pair of service frame

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 35

 Ethernet Performance Management
ITU-T Y.1731 – Technology Overview

Ethernet
Ethernet Delay Measurement Ethernet Loss Measurement Synthetic Loss
ETH-DM ETH-LM Measurement
ETH-SLM

One-Way Two-Way Single-Ended Dual-Ended Single-Ended

ETH-DM ETH-DM ETH-LM ETH-LM ETH-SLM

Scope

•  One-Way delay •  Two-Way delay •  Unidirectional frame •  Unidirectional frame •  Unidirectional frame
measurements measurements loss measurements loss measurements loss measurements
•  Also One-Way delay •  Applicable to P2P •  Applicable to P2P •  Applicable to P2P
when synchronized services only services only and MP services
Specifics

•  1DM PDU •  DMM / DMR PDUs •  LMM / LMR PDUs •  CCM PDUs •  SLM / SLR PDUs
•  Synthetic traffic with •  Synthetic traffic with •  On-demand operation •  Proactive operation •  Based on statistical
two (2) timestamps two (2) mandatory •  Based on actual •  Based on actual sampling
•  Need for Time-of-Day and two (2) optional Service Frame Loss Service Frame Loss •  Exchange of
synchronization timestamps •  Exchange of service •  Exchange of service synthetic frame
frame counters frame counters counters

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 36

Y.1731 – Frame Delay Measurement

  Frame Delay calculated Ethernet Delay Measurement

based on timestamps ETH-DM
applied to synthetic traffic
  Applicable to point-to-
point and multipoint One-Way ETH-DM Two-Way ETH-DM
services
  Two (2) mechanisms
defined Performance Metrics
Performance Metrics
One-Way ETH-DM •  One-Way FD/FDV
•  Two-Way FD/FDV
•  Time-of-Day (ToD)
•  One-Way FD/FDV when
Two-Way ETH-DM synchronization required
synchronized
for OW FD

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 37

ITU-T Y.1731 Overview
One-Way ETH-DM
Customer Service Provider Customer
Equipment Equipment

NEs must be synchronized (ToD) for one-way delay

MEP MEP

1DM
TxTimeStampf
0
1DM tp TxTimeStampf 1DM
RxTimeStampf

1DM
TxTimeStampf
0
tc
1DM TxTimeStampf 1DM
RxTimeStampf

One-Way Delay One-Way Delay Variation

Delay = RxTimeStampf – TxTimeStampf Delay Var.[tc] = Delay[tc] – Delay[tp]

tp – Time Previous 1DM – One-way Delay Measurement

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public tc – Time Current 38
 ITU-T Y.1731 Overview
Two-Way ETH-DM
Customer Service Provider Customer
Equipment Equipment

NEs may be synchronized (ToD) for one-way delay

MEP MEP

D TxTimeStampf
M 0
TxTimeStampf D
M
0 DMM RxTimeStampf M
0
0
0 M

TxTimeStampf Optional TxTimeStampf D

D
RxTimeStampf Timestamps RxTimeStampf M
M TxTimeStampb
R
TxTimeStampb DMR R
RxTimeStampb 0
Two-Way Delay One-Way Delay (Forward) One-Way Delay (Backward)
(RxTimeStampb – TxTimeStampf) –
RxTimeStampf – TxTimeStampf RxTimeStampb – TxTimeStampb
(TxTimeStampb – RxTimeStampf)
DMM – Delay Measurement Message
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public DMR – Delay Measurement Reply 39
Y.1731 – Frame Loss Measurement
  Frame Loss calculated based on
actual in-profile service counters Ethernet Loss Measurement
  Applicable to point-to-point ETH-LM
services only (with ETH-LM)
  Near-End Frame Loss
measurement
Single-Ended ETH- Dual-Ended
Loss associated with Ingress Data Frame
LM ETH-LM
  Far-End Frame Loss
measurement
Loss associated with Egress Data Frame

  Per-CoS counters maintained per

MEP Performance Metrics
TxFCl – in-profile data frames transmitted •  Unidirectional Frame Loss
towards the peer MEP •  Frame Loss Ratio
RxFCl – in-profile data frames received •  Availability
from the peer MEP

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 40

 ITU-T Y.1731 Overview
Single-Ended (On-demand) ETH-LM
Customer Service Provider Customer
Equipment Equipment

Service Frames

MEP MEP
Per-CoS
Service Frame
counters
TxFCf TxFCf
TxFCl LMM 0 LMM 0 LMM
RxFCl 0 0
tp Per-CoS
TxFCf LMR TxFCf LMR Service Frame
LMR RxFCf RxFCf counters
TxFCb TxFCb
RxFCl
LMM TxFCl
tc
LMR
Frame Loss (Far-end) Frame Loss (Near-end)
(TxFCf[tc] – TxFCf[tp]) – (RxFCf[tc] – RxFCf[tp]) (TxFCb[tc] – TxFCb[tp]) – (RxFCl[tc] – RxFCl[tp])

tp – Time Previous LMM – Loss Measurement Message

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public tc – Time Current LMR – Loss Measurement Reply41
Synthetic Loss Measurement (ETH-SLM)

  Y.1731 ETH-LM cannot be used

for frame loss in multipoint EVCs
SLR SLM
  A new protocol (SLM) based on
synthetic frames
Statistical Frame Loss
Covers P2P or MP EVCs
Implementable by ALL platforms
A B
  ITU agreed (June 2010) to
allocate code points for new
PDUs (SLM / SLR)
  ETH-SLM included ITU-T G.
8013 / Y.1731 (07/2011)
C

SLM – Synthetic Loss Message

SLR – Synthetic Loss Reply

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 42

 ITU-T Y.1731 Overview
Single-Ended ETH-SLM
Customer Service Provider Customer
Equipment Equipment

Synthetic Frames

MEP MEP
Per-Test ID
Synthetic Frame
counters
TxFCf TxFCf SLM
TxFCl SLM
0 SLM 0
RxFCl tp Per-Test ID
TxFCf SLR TxFCf SLR Synthetic Frame
SLR TxFCb counters
TxFCb
RxFCl
TxFCl
SLM
tc RxFCl = TxFCl
SLR at Responder

Frame Loss (Far-end) Frame Loss (Near-end)

(TxFCf[tc] – TxFCf[tp]) – (TxFCb[tc] – TxFCb[tp]) (TxFCb[tc] – TxFCb[tp]) – (RxFCl[tc] – RxFCl[tp])

tp – Time Previous – Start of Measurement Period SLM – Synthetic Loss Message

© 2009 Cisco Systems, Inc. All rights reserved.
tc – Public
Cisco
Time Current – End of Measurement Period SLR – Synthetic Loss Reply 43
Ethernet Local
Management
Interface (E-LMI)
Overview

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 44

Ethernet LMI
Overview

  Provides protocol and mechanisms

used for:
Notification of EVC addition, deletion or
status (Active, Not Active, Partially Active)
to CE
Communication of UNI and EVC attributes User Network Interface
to CE (e.g. CE-VLAN to EVC map)
(UNI)
CE auto-configuration UNI-C UNI-N
Notification of Remote UNI count, name and
Metro
status to CE Ethernet
  Asymmetric protocol based on CE Network
Frame Relay LMI, mainly applicable
to the UNI (UNI-C and UNI-N) E-LMI
  Specification completed by MEF:
http://www.metroethernetforum.
org/PDFs/Standards/MEF16.doc

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 45

 Ethernet
Service Activation

© 2011 Cisco and/or its affiliates. All rights reserved. 46

 Service Activation Testing (SAT)
  Issuance of ‘Birth
Certificate’ Traffic
Generation/
Reception
  Validation of Service
Configuration
Control
  Validation of SLA Plane
Signaling

Throughput
Test Methodology
Latency
Data Plane
Loss Loopback

Jitter

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 47

 SAT Concepts
Bi-directional Test Operation
Ethernet Data-Plane Loopback
Test Traffic
TG
EVC

Test Initiator Responder

Uni-directional Test Operation

Start Reply / Request

Test Traffic
TG
EVC RL

Stop Reply / Request

Test Initiator Responder
TG = Traffic Generation function
HW-accelarated (Platform-Dependent)
RL=Responder Logic

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 48

 MEF Projects

Project Name Highlights Status

• Defines set of tests needed to run before handing Approved Project

Service Activation
off a service to subscriber Editor: CableLabs
Testing project
• Leverages ITU-T Y.1564 Targeting completion in Q2, 2013

• Defines the test PDUs that can be used to perform

Approved Project
the tests defined by the SAT project
Editor: Verizon
SAT PDU project • Defines control protocol to be used for unidirectional Targeting completion in Q4, 2013
tests - to set up the collector and/or responder at the
remote end.
Approved Project
Latching loopback • Defines control protocol to be used for bidirectional Editor: Adtran
project tests - to put remote device into latching loopback Targeting completion in Q1, 2014

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 49

 ITU-T Y.1564
  RFC2544 methodology shortcomings (as
stated by Y.1564)
Not Ethernet service aware
Tests run as a single flow at a time 2544 1564
Tests are performed sequentially
Does not measure Frame Delay Variation Service centric

Does not verify CIR, CBS, EIR, EBS and CM Device Centric All KPI measure
1 test
d in

  Y.1564 advantages Sequential Tests Includes Jitter, CI

EIR,CBS/EBS
R/

Tests all KPIs at same time Missing KPI

Faster Execution

Test to CIR to verify SLA performance (FD, FDV,

FLR)
Tests to EIR limit and just beyond to verify
policing behavior (no SLA performance expected
for yellow frames (above CIR and below CIR
+EIR)

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 50

 Ethernet OAM
Interworking

Presentation_ID © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 51
What Is OAM Interworking?
Operator A Operator B

Edge Router
Edge Bridge

ATM Switch

ATM Switch
  Strict OAM layering should be honored: messages should not cross layers
Customer

Router

Router

Router
Bridge

Bridge

Bridge
switch

Customer
switch
  OAM Messages should not leak outside domain boundaries within a layer
Bridge

Bridge
  Interworking is event translations & not necessarily 1:1 message mapping
  Interworking may be inter-layer and intra-layer

Ethernet Ethernet Event

10G Translation
ATM inter-
over SONET over MPLS Ethernet RFC1483
domains intra-layer

Service OAM

Server Layers feed

events into Client Layers
Network Network
OAM OAM

Transport Transport Transport Transport

OAM OAM OAM OAM

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 52

Ethernet OAM
Deployment Use
Cases

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 53

 Ethernet OAM Deployment Use Case #1
Ethernet L2 VPN SLA Monitoring

Ethernet Service Provider

L2 VPN
Customer
L2 VPN L3 CE
EVC NID /
Customer
Access /
L3 CE
Demarcation Device
EVC

Shadow
Router NID
Optional
L2 VPN
Customer
L3 CE

EVC = Ethernet Virtual Circuit

NID = Network Interface Device
= CFM Maintenance Association End Point (MEP)
----- = IP-based probe

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 54

 Ethernet OAM Deployment Use Case #2
Mobile Backhaul Service Monitoring

Distribution
Router
Ethernet Service Provider

SAE 4G
GW
EVC Demarc
RNC 3G Probe
(Optional)
eNodeB (4G) Cell Site Demarc EVC Demarc
NodeB (3G) Router BSC 2G
BTS (2G)

Distribution
Router

Mobile Operator Wireline Provider / Mobile Backhaul Operator Mobile Operator

Cell Site(s) MTSO

eNode B = Enhanced Node B MTSO = Mobile Telephone Switching Office EVC = Ethernet Virtual Circuit
SAE GW = System Arch Evolution GW BTS = Base Transceiver Station NID = Network Interface De`vice
RNC = Radio Network Controller BSC = Base Station Controller = CFM Maintenance Association End Point (MEP)

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 55

 Ethernet OAM Deployment Use Case #3
Monitoring of Ethernet Access to L3VPN

Ethernet Service Provider

VRF
Internet

L3 VPN
Customer EVC Aggregator L3 PE
Demarc Device IP / MPLS Core
Managed NID EVC
L3 CE
Shadow
G.SHDSL Router
DSLAM Optional

Managed
L3 CE

IP Department Third Party Ethernet Service Provider IP Department

Or Transport / Aggregation Department

EVC = Ethernet Virtual Circuit

NID = Network Interface Device
= CFM Maintenance Association End Point (MEP)
----- = IP-based probe
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 56
 Ethernet OAM Deployment Use Case #4
Monitoring of Broadband & Wholesale Ethernet Access Service

Residential Ethernet Service Provider

Subscribers
VRF
Internet

EVC Aggregator BNG/

BRAS SP Retail ISP
DSLAM Aggregator EVC
PON OLT (Bitstream Wholesale)
Internet

Retail ISP
Wholesale SP
subscribers

Broadband Department Third Party Ethernet Service Provider Broadband Department /

Or Transport / Aggregation Department Wholesale SP
EVC = Ethernet Virtual Circuit
NID = Network Interface Device
= CFM Maintenance Association End Point (MEP)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 57
Ethernet OAM Deployment Use Case #5
Monitoring of Ethernet Service over DOCSIS Cable

L2 VPN
Customer

Ethernet NID /
Cable MSO Cable MSO
(HFC) Service Provider Demarcation
(HFC)
EVC

L2 VPN NID /
Customer Demarcation

L2 VPN
Customer

EVC = Ethernet Virtual Circuit

NID = Network Interface Device
= CFM Maintenance Association End Point (MEP)
----- = IP-based probe

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 58

Ethernet OAM Deployment Use Case #6
Ethernet Last Mile Testing with Dataplane Loopback

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 59

Ethernet OAM Deployment Use Case #7
Fault Management - Fault detection, Loop detection, Node monitoring,
LAG port monitoring, reachability check ….

Customer
Ethernet Service Provider

Ethernet NID /
Ethernet
NID / Transport Or MPLS Core Demarcation
Demarcation Ethernet
Transport

Customer NID /
Demarcation

Customer

EVC = Ethernet Virtual Circuit

NID = Network Interface Device
= CFM Maintenance Association End Point (MEP)
----- = IP-based probe
Link OAM Sessions

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 60

Deployment Use Case #8
Service Activation and Throughput Test with Loopback

  Application: Service Turn-Up (out-of-service)

Throughput Test - POP-Local CPE (shown) or POP-Remote CPE
Customer site Ethernet Service Provider L3 Provider POP
X,Y Central
A,B
ip = Y Test Head
mac = C
Loopback
Test Traffic
EVC EVC
VRF

ip = X ip = W
Managed mac = A mac = B
CPE L3 PE
A,B
X,Y
L3
Loopback
Hardware-accelerated (no CPU intervention)
Ethernet
Data-plane IPv4/IPv6-aware Loopback
Loopback
Configurable source IPv4/IPv6 of packets to loop
IP address swap (implicit MAC swap)

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 61

 Fault Management
Scenarios

Presentation_ID © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 62
 High Level Topology
Cisco
3800 L3 VPN
MPLS
“A”
Core
L3 VPN
“B”
Catalyst
3750-ME
CFM
Ethernet access to
IEEE 802.1ag
L2VPN P2P / MP MPLS VPN (EoMPLS)
(EoMPLS / VPLS)
MPLS
E-LMI
Aggregation

Cisco
7600 Access
Cisco
Cisco ME-3400
1800 Cisco
Link OAM
2800
IEEE 802.3ah
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 63
Fault Management Scenarios
Number Description
1 End-to-End Service/Path Verification
Fault Verification/Isolation
2 Using E-OAM for Ethernet Access to L3VPN
3 E2E CPE Fault Notification & CPE Corrective Action
using Ethernet OAM Interworking

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 64

 Operational Scenario 1
  Problem Statement
Fault Verification and Fault Isolation of ethernet connectivity issues
  Problem Solution
IEEE 802.1ag (CFM) Ping and Traceroute utilities for reactive
troubleshooting of service connectivity issues
Customer Service Provider

Ethernet MPLS Ethernet

Access Aggregation Access

CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31

Customer
Domain

Service Provider
Domain

Operator Domain

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 65

 Operational Scenario 1 (cont.)
  Problem Statement
Fault Verification and Fault Isolation of ethernet connectivity issues
  Problem Solution
IEEE 802.1ag (CFM) Ping and Traceroute utilities for reactive
troubleshooting of service connectivity issues
Customer Service Provider
uPE11 uPE31

EthernetuPE11#ping ethernet MPLS Core level 4 vlan 100

mpid 3100 Ethernet
Access Access
Type escape sequence to abort.
CE 11 uPE 11 Sending
PE-Agg nPE511Ethernet CFM loopback messages,
nPEtimeout
31 is 2 uPE
seconds:
31 CE 31
!!!!! Customer
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Domain

Service Provider
Domain

Operator Domain

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 66

 Operational Scenario 1 (cont.)
  Problem Statement
Fault Verification and Fault Isolation of ethernet connectivity issues
  Problem Solution
IEEE 802.1ag (CFM) Ping and Traceroute utilities for reactive
troubleshooting of service connectivity issues
Customer Service Provider
uPE11 nPE11 nPE31 uPE31

MPLS Core
Ethernet uPE11#traceroute ethernet 0012.017c.3d00 level 4Ethernet
vlan 100
Type escape sequence to abort. TTL 255. Per-Hop Timeout is 10 seconds
Access
Access Tracing the route to 0012.017c.3d00 on Domain PROVIDER_DOMAIN, Level 4, vlan 100
Traceroute sent via GigabitEthernet0/16
CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31
CE 31
Customer
--------------------------------------------------------------------------------
MAC Ingress Ingress Action Relay Domain
Action
Hops Host Forwarded Egress Egress Action Next Hop
Service Provider
--------------------------------------------------------------------------------
B 1 nPE11 0013.5f21.cec5 Gi3/1 IngOk Domain
RlyCCDB
Forwarded
B 2 nPE31 0007.8508.3485 RlyFDB
Forwarded Gi3/1 EgrOK uPE31
Operator Domain
! 3 uPE31 0012.017c.3d00 Gi1/1/1 IngOk RlyNone
Not Forwarded
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 67
 Operational Scenario 1 (cont.)
  Problem Statement
Fault Verification and Fault Isolation of ethernet connectivity issues
  Problem Solution
IEEE 802.1ag (CFM) Ping and Traceroute utilities for reactive
troubleshooting of service connectivity issues
nPE31(config)#int gig3/1
Customer Service Provider nPE31(config-if)#shutdown
uPE11 nPE11 nPE31 uPE31
X

Proactive Ethernet Jan MPLS Core Ethernet

26 03:14:10.608: %ETHER_SERVICE-6-EVC_STATUS_CHANGED:
changed to InActive
status of EVC_P2P_100

Access Access
Fault Notification uPE11#traceroute ethernet 0012.017c.3d00 level 4 vlan 100
CE 11 uPE 11 PE-Agg
Type nPE 11 sequence
escape nPE Timeout
to abort. TTL 255. Per-Hop 31 is 10uPE 31
seconds CE 31
Customer
Tracing the route to 0012.017c.3d00 on Domain PROVIDER_DOMAIN, Level 4, vlan 100
Reactive Traceroute sent via GigabitEthernet0/16
Domain
Fault Isolation --------------------------------------------------------------------------------
MAC Ingress Ingress Action Relay ActionService Provider
Hops Host Forwarded Egress Egress Action Next Hop Domain
--------------------------------------------------------------------------------
B 1 nPE11 0013.5f21.cec5 Gi3/1 IngOk RlyCCDB
Forwarded
B 2 nPE31 0007.8508.3485 RlyCCDB Operator Domain
Not Forwarded Gi3/1 EgrDown
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 68
 Operational Scenario 2
  Problem Statement
Troubleshooting Ethernet access connectivity problems by L3VPN PE
  Problem Solution
IEEE 802.1ag CFM with Outward-facing / Down MEPs at L3VPN PE

Customer Service Provider

Ethernet MPLS
Access Aggregation

CE 11 uPE 11 PE-Agg nPE 11 nPE 21 L3VPN PE

Customer
Domain

Service Provider
Domain

Operator Domain

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 69

 Operational Scenario 2 (cont.)
  Problem Statement
Troubleshooting Ethernet access connectivity problems by L3VPN PE
  Problem Solution
IEEE 802.1ag CFM with Outward-facing / Down MEPs at L3VPN PE
L3VPN-PE#show running-config | begin GigabitEthernet3/0/0
Customer
interface GigabitEthernet3/0/0 Service Provider
description L3VPN PE to nPE21 gig3/3
ethernet cfm mep level 4 outward domain PROVIDER_DOMAIN mpid 2450 vlan 450
ethernet cfm mep level 4 outward domain PROVIDER_DOMAIN mpid 2350 vlan 350
!
interface GigabitEthernet3/0/0.350
description To CE31
encapsulation dot1Q 350
ip vrf forwarding BLUEEthernet MPLS Core Ethernet
ip address 1.1.1.1 255.255.255.0
Access Access
!
interface GigabitEthernet3/0/0.450
CE 11 To uPE
description CE21 11 PE-Agg nPE 11 nPE 21 uPE 21 L3VPN PE
encapsulation dot1Q 450 Customer
ip vrf forwarding RED Domain
ip address 1.1.1.1 255.255.255.0
Service Provider
L3VPN-PE#show ethernet cfm maintenance-points remote Domain
Can only Ping/Traceroute to remote MEPs marked with *

MPID Level Mac Address Vlan PortState InGressPort Age(sec) Service ID Operator Domain
3350* 4 0012.017c.3d00 350 UP Gi3/0/0.350 20 customer_350_provider
2451* 4 0019.552c.0b80 450 UP Gi3/0/0.450 23 customer_450_provider
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 70
 Operational Scenario 3
  Problem Statement
End to End Fault Notification of service status to Customer Equipment
  Problem Solution
IEEE 802.3ah to IEEE 802.1ag (CFM) Inter-Working (IW) for
propagation of EVC status and CFM to E-LMI IW for notification to CE
Customer Service Provider

Ethernet MPLS Ethernet

Access Aggregation Access

CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31

Customer
Domain

Service Provider
Domain

Operator Domain

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 71

 Operational Scenario 3 (cont.)
  Problem Statement
End to End Fault Notification of service status to Customer Equipment
  Problem Solution
IEEE 802.3ah to IEEE 802.1ag (CFM) Inter-Working (IW) for
propagation of EVC status and CFM to E-LMI IW for notification to CE
Customer 4 3 Service Provider 2 1

E-LMI CFM E-LMI

Ethernet MPLS Core Ethernet
Link OAM Link OAM
Access Access

CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31

Customer
Domain

Service Provider
Domain

Operator Domain

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 72

 Operational Scenario 3 (cont.)
  Problem Statement
End to End Fault Notification of service status to Customer Equipment
  Problem Solution
IEEE 802.3ah to IEEE 802.1ag (CFM) Inter-Working (IW) for
propagation of EVC status and CFM to E-LMI IW for notification to CE
Customer 4 3 Service Provider 2 1

Ethernet MPLS Core Ethernet

Access Access

CE 11 uPE 11 PE-Agg nPE 11

CE31(config)#int fast 0/0 nPE 31 uPE 31 CE 31
CE31(config-if)#shutdown Customer
Domain
Jan 26 03:40:08.176: %ETHERNET_OAM-6-EXIT_SESSION: The client on interface Fa0/0
has left the OAM session. Service Provider
Jan 26 03:40:10.180: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to
Domain
administratively down
Jan 26 03:40:11.180: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0,
changed state to down
Operator Domain

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 73

 Operational Scenario 3 (cont.)
  Problem Statement
End to End Fault Notification of service status to Customer Equipment
  Problem Solution
IEEE 802.3ah to IEEE 802.1ag (CFM) Inter-Working (IW) for
propagation of EVC status and CFM to E-LMI IW for notification to CE
Customer 4 3 Service Provider 2 1

Jan 26 03:40:08.176: %ETHERNET_OAM-6-RFI: The client on interface Fa1/0/1 has received a remote failure
Indication from its remote peer (failure reason = remote client administratively turned off)
Ethernet MPLS Core Ethernet
Access
Jan 26 03:40:08.184: %ETHER_SERVICE-6-EVC_STATUS_CHANGED: status of EVC_P2P_100 Access
changed to InActive
Jan 26 03:40:09.191: %ETHERNET_OAM-6-EXIT_SESSION: The client on interface Fa1/0/1 has left the OAM session.
CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31
uPE31#show ethernet service evc Customer
Identifier Type Act-UNI-cnt Status Domain
EVC_P2P_100 P-P 1 InActive

uPE31#show ethernet lmi evc

Service Provider
UNI Id: CE31_UNI Domain
St Evc Id CE-VLAN
--- ---------------------------------------- ----------------------------------
?I EVC_P2P_100 100 Operator Domain

Key: St=Status,
© 2009A=Active, P=Partially
Cisco Systems, Inc. All rights reserved. Active, I=Inactive, *=Default EVC, ?=Link Down
Cisco Public 74
 Operational Scenario 3 (cont.)
  Problem Statement
End to End Fault Notification of service status to Customer Equipment
  Problem Solution
IEEE 802.3ah to IEEE 802.1ag (CFM) Inter-Working (IW) for
propagation of EVC status and CFM to E-LMI IW for notification to CE
Customer 4 3 Service Provider 2 1

Jan 26 03:40:36.093: %ETHER_SERVICE-6-EVC_STATUS_CHANGED: status of EVC_P2P_100 changed to InActive

Ethernet
uPE11#show ethernet MPLS Core
cfm maintenance-point remote Ethernet
Access Vlan PortState InGressPort
MPID Level Mac Address Age(sec) ServiceAccess
ID
3100 4 0012.017c.3d00 100 DOWN Gi0/16 21 customer_100_provider
CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31
uPE11#show ethernet service evc Customer
Identifier Type Act-UNI-cnt Status Domain
EVC_P2P_100 P-P 1 InActive

uPE11#show ethernet lmi evc

Service Provider
UNI Id: CE11_UNI Domain
St Evc Id CE-VLAN
--- ---------------------------------------- ----------------------------------
I EVC_P2P_100 100 Operator Domain

Key: St=Status, A=Active,

© 2009 Cisco Systems, P=Partially
Inc. All rights reserved. CiscoActive,
Public I=Inactive, *=Default EVC, ?=Link Down 75
 Operational Scenario 3 (cont.)
  Problem Statement
End to End Fault Notification of service status to Customer Equipment
  Problem Solution
IEEE 802.3ah to IEEE 802.1ag (CFM) Inter-Working (IW) for
propagation of EVC status and CFM to E-LMI IW for notification to CE
Customer 4 3 Service Provider 2 1

Ethernet MPLS Core Ethernet

Access Access
CE11#show ethernet lmi evc
UNI Id: CE11_UNI
CE E2E
Proactive 11 uPE 11 St
PE-Agg
Evc Id
nPE 11 nPE 31
CE-VLAN
uPE 31 CE 31
Customer
--- ---------------------------------------- ----------------------------------
Fault Notification I EVC_P2P_100 100 Domain

ServiceDown
Key: St=Status, A=Active, P=Partially Active, I=Inactive, *=Default EVC, ?=Link Provider
Domain
Proactive CPE CE11#show ip interface brief
Interface IP-Address OK? Method Status Protocol
Action <snip> Operator Domain
GigabitEthernet0/0.100 100.100.100.11 YES manual down down
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 76
Summary

  You can perform fault, performance and configuration

management with E-OAM for Ethernet based Services
  There are multiple suites of protocols from various
standard bodies (& vendors) that work in different
bucket
  And they do interwork to give you a comprehensive
troubleshooting and maintenance platform.

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 77

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 78
Acronyms
Acronym Acronym
AIS Alarm Indication Signal MEF Metro Ethernet Forum
CCM Continuity Check Message MEN Metro Ethernet Network
CCMDB CCM Data Base (see CCM) MEP Maintenance Association End Point
CE Customer Edge MEPID MEP Identifier (see MEP)
CFM Connectivity Fault Management MHF MIP Half Function (see MIP)
EFM Ethernet in the First Mile MIB Management Information Base
E-LMI Ethernet LMI (see LMI) MIP Maintenance Domain Intermediate Point
E-OAM Ethernet OAM (see OAM) MP Maintenance Point
EVC Ethernet Virtual Connection Operations, Administration and
OAM Maintenance
Institute of Electrical and
IEEE Electronics Engineers PDU Protocol Data Unit
ITU International Telecommunication Union PE Provide Edge
LBM Loopback Message RDI Remote Defect Indicator
LBR Loopback Reply RFI Remote Failure Indicator
LMI Local Management Interface TLV Type, Length, Value
LTM Linktrace Message UNI User to Network Interface
LTR Linktrace Reply UNI-C Customer side of UNI (see UNI)
MA Maintenance Association UNI-N Network side of UNI (see UNI)
MAID MA Identifier (see MA) VID VLAN Identifier
MD Maintenance Domain VLAN Virtual LAN

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 79