CiTRANS 660 Troubleshooting Guide
CiTRANS 660 Troubleshooting Guide
Troubleshooting Guide
Version: A
Code: MN000001394
December 2012
Thank you for choosing our products.
Related Documentation
Document Description
I
Document Description
II
Version
Version Description
A Initial version.
Intended Readers
u Commissioning engineers
u MPLS-TP technology
u Ethernet technology
III
Conventions
Terminology Conventions
Terminology Convention
FiberHome CiTRANS 660 High-capacity MPLS-TP
CiTRANS 660
Platform
FiberHome e-Fim OTNM2000 Element Management
OTNM2000
System
IV
Symbol Conventions
Cascading
→ Connects multi-level menu options.
menu
Bidirectional
↔ The service signal is bidirectional.
service
Unidirectional
→ The service signal is unidirectional.
service
V
Contents
Preface...................................................................................................................I
Version ..........................................................................................................III
Conventions ................................................................................................. IV
1 Overview of Troubleshooting...........................................................................1
2 Troubleshooting Flow....................................................................................17
5.2 All NEs in a Certain Network Block or the Entire Network Out of
Management ..................................................................................81
11.2 All Services Interrupted due to Line Card Failure ........................... 114
Appendix C Abbreviations..........................................................................126
Figures
Figure 1-1 Positions of the ESD protection earth ground fastener, subrack earth
ground pole ......................................................................................3
Figure 1-2 Slide rails..........................................................................................5
Figure 1-3 Dedicated fiber puller ........................................................................6
Figure 1-4 No access to Internet ........................................................................9
Figure 1-5 Do not modify IP addresses ............................................................10
Figure 1-6 Do not modify protocol settings ....................................................... 11
Figure 1-7 Do not modify computer name.........................................................12
Figure 1-8 Do not modify LAN settings .............................................................13
Figure 2-1 Troubleshooting flow.......................................................................19
Figure 3-1 Network diagram - layer-by-layer method ........................................26
Figure 3-2 Network diagram - block-by-block method .......................................28
Figure 3-3 Diagram of section-by-section method.............................................29
Figure 3-4 Network diagram - section-by-section method .................................31
Figure 3-5 Alarm and performance analysis .....................................................34
Figure 3-6 Hardware loopback (optical interface)..............................................36
Figure 3-7 Signal flow of line loopback .............................................................37
Figure 3-8 Signal flow of equipment loopback ..................................................38
Figure 3-9 Card connection under loopback method ........................................39
Figure 3-10 Network diagram - substitution method............................................41
Figure 3-11 Meter test method ...........................................................................43
Figure 4-1 Installing a card...............................................................................48
Figure 4-2 Unplugging a card...........................................................................49
Figure 4-3 Unplugging the SFP / XFP optical module .......................................55
Figure 4-4 Installing the SFP / XFP optical module ...........................................55
Figure 4-5 The snap-in latch of the fan unit.......................................................57
Figure 4-6 Removing the fan unit .....................................................................58
Figure 4-7 Installing the fan unit .......................................................................59
Figure 4-8 Removing the subrack's anti-dust screen ........................................60
Figure 4-9 Installing the subrack's anti-dust screen ..........................................60
Figure 4-10 Hardware loopback (optical interface)..............................................62
Figure 4-11 Hardware loopback (Ethernet interface) ..........................................62
Figure 4-12 Hardware loopback (E1 interface) ...................................................63
Figure 4-13 Connection of optical Tx power test .................................................68
Figure 4-14 Connection of optical Rx power test.................................................70
Figure 4-15 Querying current alarms..................................................................73
Figure 4-16 Querying alarm history ....................................................................74
Figure 4-17 Querying current performance.........................................................76
Figure 4-18 Querying performance history .........................................................77
Figure 8-1 Troubleshooting flow of CES interruption .........................................92
Figure 8-2 Troubleshooting flow of Ethernet service interruption .......................95
Figure 9-1 Troubleshooting flow of QoS abnormality ......................................100
Figure 10-1 Troubleshooting flow of the network level protection switching
failure ...........................................................................................104
Figure 11-1 Network topology of tributary card failure .......................................108
Figure 11-2 Example of tributary card failure - VP layer loopback frame
configuration................................................................................. 110
Figure 11-3 Example of tributary card failure - VP layer CV frame configuration of
NE1.............................................................................................. 111
Figure 11-4 Example of tributary card failure - VC layer loopback frame
configuration................................................................................. 112
Figure 11-5 Example of tributary card failure - VC layer CV frame configuration 113
Figure 11-6 Network topology of line card failure .............................................. 115
Figure 11-7 Example of line card failure - VS loopback frame configuration ...... 116
Figure 11-8 Network topology of line failure...................................................... 117
Tables
During troubleshooting of the equipment, the maintenance staff should follow the
operation safety rules, so as to avoid bodily injury and equipment damage. In
addition, the maintenance staff should master certain technical knowledge and
operation skills.
Version: A 1
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
The maintenance staff should be familiar with the safety and warning signs, whose
meanings are shown in Table 1-1 and Figure 1-1.
2 Version: A
1 Overview of Troubleshooting
(1) ESD protection sign (2) ESD protection earth ground fastener
(3) Subrack earth ground sign (4) Subrack earth ground pole
Figure 1-1 Positions of the ESD protection earth ground fastener, subrack earth ground pole
Relevant knowledge
Version: A 3
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Note:
Safety measures
To wear an ESD protection wrist, put one end on your wrist with the metal fastener
on it in close contact with the skin, and fasten the other end to the ESD protection
earth ground fastener on the right mounting ear of the subrack.
u Before plugging / unplugging a card, you need to wear ESD protection gloves
or the ESD protection wrist strap, and keep your hands dry and clean.
u When taking a card, do not touch the devices and wiring troughs on it.
4 Confirm that this card is not connected with wires and cables or optical
fibers.
4 Confirm the insertion direction of this card, and do not insert the card in the
opposite direction. The side with the card name should be placed
downward.
4 Operate the card gently to avoid distorting pins on the subrack backplane.
4 Slide the card gently along the slide rails to avoid distorting pins on the
backplane. The slide rails are shown in Figure 1-2.
4 Version: A
1 Overview of Troubleshooting
Version: A 5
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Optical fibers are easily damaged if they are plugged or unplugged by hand without
using any tool.
Caution:
The fiber puller, as the delivery accessory, is similar to tweezers in appearance and
attached with a spring cord, as shown in Figure 1-3.
The fiber puller has been installed on the cabinet with the spring cord before delivery.
Clip the fiber connector with the fiber puller to plug and unplug fibers.
(1) Spring cord (2) Connected to the installation (3) Fiber puller
hole on the vertical mounting
flange
Do not insert the pigtail into the card’s optical interface when the pigtail’s Tx optical
power is unknown. To avoid damage to optical modules, just put the pigtail in the
card’s optical interface without connecting them or add an attenuator.
6 Version: A
1 Overview of Troubleshooting
u Check the optical power before connecting fibers. Connect the optical fibers
only when the optical power meets the power requirements of the optical
interfaces. See product description for optical power specifications of all rates
of optical modules.
u Before connecting optical fibers, check whether the optical fiber connector
matches the optical interface. If the optical fiber connector does not match the
optical interface, use a compatible connector.
Protecting eyes
High optical power can cause bodily harm, especially to eyes. Never look directly
into the end of the optical transmitter fiber jumper or the end of its active connector
Exercise care if you must bend fibers. If bends are necessary, the fiber bending
radius should never be less than 38mm.
u Cover the optical interfaces of the replaced cards with anti-dust caps to keep
the optical interfaces clean.
Cleaning
While cleaning fiber connectors or active connectors, you should use dedicated
cleaning tools and materials.
The following lists some common tools; the maintenance staff can choose to
prepare some of them.
u Dedicated cleaning solvent (the first choice is the isoamyl alcohol and the
second choice is the isopropyl alcohol )
Version: A 7
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Shorting
u When a short circuit occurs, a quick, significant rise in the short circuit current
will easily result in damage to the equipment and unforeseen safety problems.
u During operation, avoid metal fillings, water and other conductive materials into
the in-service equipment so as to prevent damage to electrical apparatus and
components caused by a short circuit.
Grounding
u Ensure that the protection earth ground bar in the equipment room is securely
connected.
u Before removing the power cable, confirm that the power supply is
disconnected.
u The power cable should not be exposed. The exposed part should be
completely covered with insulating tape.
8 Version: A
1 Overview of Troubleshooting
u If the operation condition permits, shut down the power first and then perform
other operations.
u Use UPS power supply to avoid loss of network management data caused by
accidental power failure.
u To shut down the OTNM2000 server, first exit the operation system normally
and then shut off the power supply.
u Do not delete any file in the network management system randomly or copy
any irrelevant file into the OTNM2000 server.
u Do not visit Internet via the OTNM2000 server. Doing so may increase data
flow in the net card and hence affects normal network management data
transmission or results in other accidents.
Version: A 9
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
10 Version: A
1 Overview of Troubleshooting
Version: A 11
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
12 Version: A
1 Overview of Troubleshooting
u To avoid illegal operations by lower-level users, the higher-level users can use
the authorization and domain division function to divide the network into
different management domains, and assign the management authorization of
certain domains for lower-level users.
Version: A 13
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
u If you need to modify the service configuration, keep the following rules in mind:
4 Before modifying the configuration, read the NE data from the equipment,
so as to ensure that the configuration data of the OTNM2000 and the NE
are the same; at the same time, back up the network management data,
for restoring the data when accidents occur.
Note:
Exiting the network management system does not interrupt traffic in the
network, but precludes centralized control of the networked equipment.
14 Version: A
1 Overview of Troubleshooting
Be familiar with the network information of the project to facilitate the fault analysis
and isolation.
u Be familiar with the network condition of the project, including the distance
between stations, station type and card configuration.
u Be familiar with the service configuration, the optical fiber wiring conditions at
the ODF (Optical Distribution Frame), and the arrangement of equipment sets
in the equipment room of each station.
u Be familiar with the operating status of the equipment at the local office.
Being acquainted with the basic operations on the equipment helps you to handle
faults quickly.
When handling faults, the maintenance staff needs to use tools and meters. List of
Tools and Meters lists the common tools and meters. Being familiar with the
application of tools and meters helps you to perform the fault analysis and handling
quickly.
Version: A 15
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
The maintenance staff should master the operation methods to collect and save the
field data, so as to restore the normal service communication quickly using the
backed-up data when loss of the network management data causes service
interruption; in addition, the field data can help analyze and isolate faults when the
equipment has faults.
For the operation methods to collect and save the field data, please refer to related
chapters and sections in routine maintenance.
Common tools and meters used in troubleshooting are listed in Table 1-2.
Optical power meter Tests the optical interface’s Tx and Rx optical power.
Variable optical attenuator Reduces optical power reduction to protect the optical interface
(VOA) against over-high optical power
Measures the voltage drop of the cable terminal pairs at the the
Cable tester
maximum rated current.
16 Version: A
2 Troubleshooting Flow
The following discusses the general troubleshooting process, with which the users
can apply the related knowledge in isolating and eliminating network faults quickly
and accurately, so as to solve the common faults in various network scenarios. The
following includes these contents:
Troubleshooting Flow
Flow Description
Version: A 17
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
When a fault occurs in the system, the maintenance staff can isolate the fault
quickly or seek help promptly to restore the service according to Figure 2-1.
18 Version: A
2 Troubleshooting Flow
Version: A 19
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
When handling a fault, first record the symptom of the fault (such as the related
alarms, performance events, and switching data in the OTNM2000). At the same
time, back up the network management data to facilitate the data restoration when
the fault handling operations are incorrect or fail.
Ascertain whether the fault is caused by external factors; external factors include
the power supply problem, the optical cable problem, the equipment room
environment problem (such as the temperature and humidity), and the user
equipment problem. If so, perform related troubleshooting operations accordingly.
u To ascertain the fault of this equipment, analyze and isolate the causes of the
fault according to Basic Principles and Methods for Fault Isolation and then
handle the fault using the corresponding troubleshooting methods.
u Do not make blind attempts to troubleshoot: If the fault is hard to isolate, please
send the related records of the fault to the FiberHome local representative or
the FiberHome technical support center (see the back cover of this manual for
the contact methods). Before the FiberHome maintenance staff provides
remote or on-site guide, do not make blind attempts to troubleshoot so as to
avoid exacerbating the fault.
u Restore services at first when a fault occurs: For faults such as the service
interruption, enable the protection channels promptly. If no protection channel
exists, use other grooming schemes, so as to restore services as soon as
possible.
After the fault is eliminated, observe and record the operating status of the
equipment, so as to confirm that the equipment runs normally.
20 Version: A
2 Troubleshooting Flow
Make records of the troubleshooting procedure (including the card replacement, the
software upgrade, the data modification, etc.), for convenience of future
maintenance and accumulation of maintenance skills.
Version: A 21
3 Basic Principles and Methods for
Fault Isolation
The key of troubleshooting is fault isolation. To isolate the fault point quickly and
exactly, the maintenance staff should be familiar with common methods of fault
isolation and use them comprehensively in the actual applications. The following
includes these contents:
22 Version: A
3 Basic Principles and Methods for Fault Isolation
It is significant to isolate a fault accurately since the stations are distantly apart. After
isolating the fault accurately, you can take corresponding measures to remove the
fault.
The general principles for fault isolation are: external factors first and internal factors
later, network first and NE later, higher rate first and lower rate later, higher level first
and lower level later, common first and specific later.
When isolating a fault, users should first exclude the possible external faults,
such as broken fibers, faulty trunk cables, fault in connection with the
equipment and power supply fault.
When isolating a fault, users should first isolate the fault to a single station as
accurately as possible.
The alarm signal flow shows that the higher rate signal alarms usually cause
the lower rate signal alarms. Therefore, users should eliminate any fault in
higher rate signals first.
Analyze higher level alarms such as urgent alarms and major alarms first, and
lower level alarms such as minor alarms and common alarms later. Classify the
associated alarms with different priority, and handle them respectively.
Version: A 23
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
In troubleshooting, it is the first step to isolate the fault accurately and quickly.
u Layer-by-layer method
u Block-by-block method
u Section-by-section method
u Loopback
u Substitution
u Meter test
Users can select from the above isolation methods for trouble shooting according to
the field situation and condition, and also can employ the methods together.
Overview
Data are transmitted based on the physical layer, data link layer, and network layer
in the OSI (Open System Interconnection) model. Using the layering features of the
network properly, the maintenance staff can isolate faults correctly and rapidly.
The functions, common faults, and possible causes of various layers are described
as follows:
4 The connection faults of the service fibers / cables and clock cables;
u Data link layer: Implements physical addressing and data framing. The
common faults of the data link layer include
24 Version: A
3 Basic Principles and Methods for Fault Isolation
4 The data encapsulation modes in the same data link layer are not the
same. The possible causes include
4 The routing protocol has faults. The possible causes are described as
follows:
Caution:
Application
The following explains the application of the layer-by-layer method via an example
of fault.
Version: A 25
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
As shown in Figure 3-1, the routing protocols such as the static routing protocol are
running in the network. The maintenance staff finds that the route between P1 and
P2 is not reachable, but other protocols all run normally. Under this condition,
according to the principles of the layer-by-layer method, we can ascertain that the
fault point is at the network protocol; later, via troubleshooting operations, we find
that the incorrect modification of the route configuration by the maintenance staff
causes the fault.
Overview
The OTNM2000 provides the configuration block GUI via clearly demarcated
models. This function enables the maintenance staff to classify the faults according
to the configuration block types when handling faults. The purpose of this function is
to narrow the range of fault isolation and isolate and handle faults rapidly.
26 Version: A
3 Basic Principles and Methods for Fault Isolation
Application
The following explains the application of the block-by-block method via an example
of fault.
Version: A 27
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
In the network shown in Figure 3-2, after the configuration files are delivered, the
route between R1 and R2 is not reachable. Under this condition, via performing fault
isolation using the block-by-block method, the maintenance staff can ascertain the
possible causes of the fault.
28 Version: A
3 Basic Principles and Methods for Fault Isolation
Overview
4 Service card: The possible fault causes include interface faults, card faults,
and the fault of poor contact between the card and the backplane.
4 The XCUJ2 card: The possible fault causes include external clock faults,
card faults, and the fault of poor contact between the card and the
backplane.
Version: A 29
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
4 Line card: The possible fault causes include interface faults, card faults,
and the fault of poor contact between the card and the backplane.
u Client side network: the subscriber network equipment connected with the
service card at the CiTRANS 660's client side, the cable connection between
the subscriber equipment and the service card.
u Line side network: the transmission network equipment connected with the
interface card at the CiTRANS 660's line side, the optical fiber connection
between the transmission equipment and the service card.
Note:
30 Version: A
3 Basic Principles and Methods for Fault Isolation
Application
As shown in Figure 3-4, packet loss exists in the subscriber side services. Via
testing by the meters, we ascertain that the client side and line side network of R1
both operate normally. So we can narrow the range of equipment sets and cards
related to the fault, and ascertain that the fault point is inside NE R1. Check whether
the GSJ2 card, the XCUJ2 card, and the XSJ2 card operate normally section by
section. After the GSJ2 card is unplugged and inserted again, the packet loss
disappears. Thus we can ascertain that the cause of the fault is the poor contact
between the GSJ2 card and the backplane.
Version: A 31
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Overview
When the system is faulty, a large number of alarms and performance events will be
generated. By analyzing the information, you can ascertain the fault type, narrow
down the search scope and save the time for fault isolation.
You can obtain the alarms and performance information in the following two ways:
Query the current alarms, alarm history, current performance events, and
performance history of the system via the OTNM2000. See Viewing
Performance and Viewing Alarms for detailed operations.
4 Cabinet: Three indicator LEDs in different colors (red, yellow, and green)
are provided on the cabinet top. Solid red indicates an urgent alarm of the
equipment; solid yellow indicates a non-urgent alarm; and solid green
indicates input of power supply or a service call.
4 Card: Each card has the working status LEDs on its panel with different
indications. See Indicator LEDs. By virtue of the LEDs, you can ascertain
the card operation status, the alarm levels, causes of the fault, and take
measures accordingly.
Caution:
3. The indicator LEDs will not indicate the filtered alarms set on the
OTNM2000.
32 Version: A
3 Basic Principles and Methods for Fault Isolation
The two methods to obtain alarm information should be used together in practice.
Each station cooperates with each other and is managed by the OTNM2000.
Table 3-1 compares the two ways of obtaining information.
Table 3-1 Comparison between the two ways of obtaining information (by the OTNM2000 and
by the equipment indicator LEDs)
Application
The following explains how to apply the alarm and performance analysis in a
specific example.
The network of a project is as shown in Figure 3-5. Multiple E-Line services are
established between NE1 and NE2. Ports FE1 and FE2 are used as the service
access ports. GE1 port of NE1 and GE2 port of NE2 are used as optical line
interfaces, respectively.
Version: A 33
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Fault symptom: The service between the FE1 ports of NE1 and NE2 is interrupted.
The FE1 port of NE2 reports a LINK_LOS alarm, and the current performance data
of this port indicates no packet is received.
Fault analysis: Multiple E-Line services are established between NE1 and NE2.
Only the service accessed via the FE1 ports is interrupted whereas services
accessed via other ports are normal. The optical line must be normal. That the FE1
port of NE2 reports a LINK_LOS alarm indicates that the Rx signal is lost at the FE1
port.
u The cable connection between the user equipment 3 at NE2 side and the FE1
port is faulty or the cable itself is faulty.
1. Check whether the cable between the FE1 port of NE2 and the user equipment
3 is loose. If so, secure the cable to see whether the alarm is removed. If the
alarm persists, replace the cable.
34 Version: A
3 Basic Principles and Methods for Fault Isolation
2. If the alarm still exists, check whether the user equipment 3 at NE2 side and
the NE2 equipment are faulty using the substitution method.
If the network, service and fault information is complex, a large number of alarms
and performance events will be generated following the fault for the maintenance
personnel to analyze and determine. In this case, maintenance personnel can resort
to other methods to isolate the fault and eliminate the fault with reference to the
alarms and performance events.
Some faults may not generate visible alarms or performance events (or even
generate no alarm or performance event at all). In this case, you need to isolate the
fault with other methods.
Overview
Loopback is the most commonly used method for fault isolation. After isolating the
fault to a certain scope by the alarm and performance analysis, users can further
isolate it to a certain station or card using the loopback.
Caution:
Version: A 35
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
u Hardware loopback
Perform the loopback for the physical port (optical interface and electrical
interface) using pigtail fibers and loopback cables manually.
As shown in Figure 3-6, perform the signal loopback for the optical interface or
electrical interface at the client side of NE2 using pigtails or loopback cables, so
as to connect two service flows in series to facilitate the service test.
Compared with the software loopback, the hardware loopback is more thorough
since it loops back the signals at a port directly. However, it is inconvenient to
operate and should be performed by on-site maintenance staff. See Performing
Hardware Loopback for the operation procedures of hardware loopback.
Note:
u Software loopback
36 Version: A
3 Basic Principles and Methods for Fault Isolation
Software loopback refers to the loopback on the card performed via the
OTNM2000. The signal received at the Rx port of the local equipment is looped
back to the Tx end, so as to perform the loopback of the remote end equipment.
Using this method, the maintenance staff can perform the loopback operations
more conveniently and isolate the fault more quickly. However, the range and
position of faults isolated by software loopback is less accurate than that
isolated by hardware loopback. For example, when you perform a single station
test, if the service is tested to be normal by the equipment loopback at the
optical interface, t cannot be ascertained that the optical interface card is
normal since the equipment loopback does not involve the optical module.
However, if the service is tested to be normal after the loopback of the optical
interface using a pigtail fiber, it can be ascertained that the optical interface card
is normal.
4 Line loopback
Loops back the signal received at the physical port to the corresponding Tx
port for output without changing the signal structure.
See Figure 3-7 for the line loopback signal flow of the equipment card.
4 Equipment loopback
Loops back the processed signal received at the physical port to the
corresponding Tx port .
Version: A 37
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
See Figure 3-8 for the equipment loopback signal flow of the equipment
card.
Caution:
The equipment loopback will disable the DCC of the optical interface.
Performing the equipment loopback on the tributary link will make the
equipment at the local station and the succeeding stations out of the EMS
management. In this case, users should restore the DDC by resetting
optical cards at stations manually. Always be cautious during operation.
Application
The following describes the application of the loopback method with an example.
38 Version: A
3 Basic Principles and Methods for Fault Isolation
Fault analysis: The service signal received by NE1 from NE2 has an error. Among
the current alarms on the OTNM2000, we can find an LOF alarm on NE1 and an
E1_RDI alarm on NE2.
According to the analysis above, we can find the possible fault causes including
After listing the possible causes of the fault, eliminate the causes one by one:
Procedures of troubleshooting:
1. Check the E1 cable connection. Reconnect the cable to see whether the alarm
disappears.
2. If the alarm persists, perform loopback on the ESJ1 card of NE1. If the error
disappears after the loopback, perform loopback on other cards to narrow down
the fault scope.
The loopback method can save the time of analyzing the alarm and performance
events and isolate the fault to the specific station or card quickly. It is easy for
maintenance personnel to operate.
Version: A 39
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
However, the loopback method will interrupt all the normal services in the loopback
path. This method is recommended to use only when severe faults (such as service
interruption) occur.
3.2.6 Substitution
Overview
The substitution method means to isolate and handle the fault by replacing a
possible faulty component with a normal one. The component can be a cable, a
card, or a set of equipment.
Note:
Application
The following explains the application of the substitution method via an example of
isolating the card fault.
40 Version: A
3 Basic Principles and Methods for Fault Isolation
As shown in Figure 3-10, the E-Line services between R1 and R2 are interrupted.
Use the substitution method to replace the possibly faulty cards or cables related to
the interrupted services in R1 and R2 step by step, until the faulty card or cable is
found. Then eliminate the fault.
The substitution method is simple and useful, with fewer technical requirements for
the maintenance staff. It is generally used in the case when the fault is narrowed
down to a certain scope but is not yet isolated to a specific card, device or cable.
However, this method requires spare parts at hand. Moreover, the operations are
not as convenient as other methods. If users do not operate according to the
operation specifications, problems such as the card damage may occur.
Version: A 41
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Overview
The meter test method is generally used to isolate the external faults of the
equipment and the faults of the connection with other equipment sets. The
commonly used test meters include the optical power meter, the TDM error detector,
the multimeter, the SmartBits analyzer and the cable detector.
The optical power meter is used to measure the Tx and Rx optical power of a
card.
The error detector is used to test the errors in a transmission channel, including
the errored second, the bit error ratio, etc.
Generally, users should operate as follows: Loop back the channel to be tested,
generate pseudo random codes with the error analyzer, and check the error
condition of the tested channel by the error analyzer.
u Multimeter
u SmartBits analyzer
The data analyzer is used to test and analyze the Ethernet services of layers 2
and 3 and the performance of network. It has a wide test range, and the
parameters that can be tested by it include maximum wire speed, data traffic,
frame length, throughput, packet loss rate, and network delay.
u Cable tester
The cable tester is used to measure the voltage drop of the cable terminal pair
under the maximum rated current, so as to ascertain the connectivity and the
transmission quality of the cable.
Application
The following explains the application of the meter test method via an example of
isolating the fault with the SmartBits analyzer.
42 Version: A
3 Basic Principles and Methods for Fault Isolation
In the process of commissioning, a large number of packets are lost in the client
side services of a certain set of the equipment. We use the SmartBits analyzer to
connect with the GSJ2 card at the client side and observe the data receiving
conditions of the meter, and no packet is lost. So we can exclude the faults at the
transmission equipment side. Combining the troubleshooting operations with the
block-by-block method, we can ascertain that the inconsistency between settings of
the operating modes for client side service ports causes the packet loss. Then we
set the port operating modes correctly, and the fault disappears.
The meter test method is persuasive in the isolation and analysis of faults.
However, it needs meters and has a tougher requirement of the maintenance staff;
in addition, the services on the tested channel will be interrupted.
Version: A 43
4 Common Operations of
Troubleshooting
Resetting a Card
Replacing a Component
Performing Loopback
44 Version: A
4 Common Operations of Troubleshooting
The CiTRANS 660 provides multiple methods for resetting the card software and
card hardware. See Table 4-1.
Caution:
Because the card reset operations can influence services on a card and
the communication with the network management system, do not perform
the reset operations randomly.
Operation procedures
1) Press the RESET key on the card (this key is a spring return button and it
automatically releases after being pressed).
Version: A 45
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
2) Observe the ACT indicator LED on the card. If the indicator LED blinks
again after being extinguished for several seconds, the card is restarted
and electrified normally.
Prerequisites
u The OTNM2000 can normally manage, configure, and monitor the entire
network.
u The user has obtained the network management authority with an intermediate
level or above.
The OTNM2000
Operation procedures
1. In the OTNM2000 GUI, click the NE which the card belongs to and access the
NE window.
4 In the subrack view of the NE, right-click the card, and click "Control
Command in the shortcut menu that appears; then the control command
window of this card will appear.
4 In the subrack view of the NE, click the card, and click "Control Command
in Task Panel on the right to bring up the control command window of this
card.
3. In the control command window, click the button at the right side of the reset
option. In the dialog box that appears, click "Yes to complete the issuing of the
reset command.
46 Version: A
4 Common Operations of Troubleshooting
Note:
After the operations are completed, the command manager window will
appear. It prompts whether the command is issued successfully (
represents the issuing failure, and represents the issuing success).
Warning:
Do not plug or unplug a card with excessive force. Plugging / unplugging
an in-service card may influence normal operating of the equipment and
even cause service interruption.
u When replacing a card, users must plug / unplug the card correctly, so as to
avoid damaging the equipment.
u Hot insertion of the AIFJ1 / AIFJ2 card is strictly prohibited. When plugging /
unplugging the AIFJ1 / AIFJ2 card, users must shut off the corresponding
power supply output switch of the external power supply device.
u If a single set of equipment is configured with only one AIFJ1 / AIFJ2 card,
replacing this card can cause service interruption. Under this condition, it is
recommended that you conduct the replacement operations at night.
Prerequisites
None.
Version: A 47
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Operation procedures
1. Put on the ESD protection wrist strap (ensure that the plug is inserted in the
ESD protection earth grounding fastener correctly).
2. Inserting a card
2) Align the upper and bottom edges of the card with the slide rails inside the
slot (with the device face to the right side), and slowly push it in along the
slide rails, as shown in ① of Figure 4-1.
3) Push the card to its position until the card is inserted in its socket on the
subrack.
3. Unplugging a card
1) Remove the fibers and cables that are connected with the card.
2) Hold the card panel’s upper and bottom latches using both hands, as
indicated by ① of Figure 4-2.
48 Version: A
4 Common Operations of Troubleshooting
3) Open and extend the latches to detach the card from the subrack, as
indicated by ② of Figure 4-2.
4) Hold the latches and draw out the card slowly along the slide rail, as
indicated by ③ of Figure 4-2.
During the troubleshooting process of the equipment, users generally use the
substitution method; that is, replace the suspected component with a known good
one. The components that are confirmed to have faults or need to be upgraded
should be replaced. The following discusses common operations of component
replacement, and includes these contents:
Version: A 49
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Caution:
Prerequisites
u The spare card of the same type as the card to be replaced is prepared.
u To replace the AIFJ1 / AIFJ2 card, first disconnect the external power supply for
the card.
Warning:
Do not plug / unplug the AIFJ1 / AIFJ2 card when the equipment is
running.
u The OTNM2000
1. Select a spare card: select a spare card that has the same card name, number
and software version as the card to be replaced. As for the NMUJ1 card, make
sure the DIP switches on the spare card are identical with DIP switches on the
card to be replaced as displayed on the OTNM2000.
2. Query and record the abnormal alarms and performance of the card to be
replaced.
50 Version: A
4 Common Operations of Troubleshooting
4. Wear the ESD protection wrist strap (make sure that one end of the wrist strap
is connected to the ESD fastener of the cabinet correctly).
5. Remove the wires and cables connected to the panel of the card to be replaced
(put the anti-dust cap onto the removed optical fiber), and unplug the card. For
detailed operations, see Plugging and Unplugging a Card.
6. Put the replaced card into the ESD protection bag, and attach a maintenance
label to it. Then send it back to FiberHome headquarters according to the
address on the back cover of this manual.
7. Insert the new card and re-connect the fibers and cables. For detailed
operations, see Plugging and Unplugging a Card.
8. Observe the indicator LEDs on the new card. It takes three to four minutes for
the new card to enter into normal work. See Indicator LEDs for the status of
indicator LEDs of a card in normal work.
9. Issue the Download Device Config command to the NE to which the new card
belongs via the OTNM2000. As for the NMUJ1 card, first issue the
management configuration to the network block to which the new card belongs.
10. Obtain the status, performance and relevant data of the new card via the
OTNM2000.
11. Check the card via the OTNM2000 to make sure that the alarms caused by
the card fault is removed, the performance is in normal status and no abnormal
alarm or performance event occurs.
1. Select a spare card: select a spare card that has the same card name, number
and software version as the card to be replaced. As for the NMUJ1 card, make
sure the DIP switches on the spare card are identical with DIP switches on the
card to be replaced as displayed on the OTNM2000.
2. Query and record the abnormal alarms and performance of the card to be
replaced.
Version: A 51
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
4 If the status of the faulty card is Activated, issue the card activation
command to the card to be replaced in the control command GUI, so as to
switch the services carried on it to the protection card.
Note:
If the NMUJ1 card is faulty, check whether the faulty card is in protection
status. If it is in working status, you should perform the active / standby
switch.
4 When the faulty card is in the Inactivated status, remove the fibers and
cables connected to the panel of the card and unplug the card. For
detailed operations, see Plugging and Unplugging a Card.
6. Put the replaced card into the ESD protection bag, and attach a maintenance
label to it. Then send it back to FiberHome headquarters according to the
address on the back cover of this manual.
7. Insert the new card and re-connect the fibers and cables. For detailed
operations, see Plugging and Unplugging a Card.
8. Observe the indicator LEDs on the new card. It takes three to four minutes for
the new card to enter into normal work. See Indicator LEDs for the status of
indicator LEDs of a card in normal work.
9. Issue the Download Device Config command to the NE to which the new card
belongs via the OTNM2000. As for the NMUJ1 card, first issue the
management configuration to the network block to which the new card belongs.
10. Issue the card activation command to the spare card in the control command
GUI to switch services carried by it to the new card.
11. Obtain the status, performance and relevant data of the new card via the
OTNM2000.
12. Check the card via the OTNM2000 to make sure that the alarms caused by
the card fault is removed, the performance is in normal status and no abnormal
alarm or performance event occurs.
52 Version: A
4 Common Operations of Troubleshooting
When the SFP (Small Form-factor Pluggable transceiver) / XFP (10-Gigabit small
Form-factor Pluggable transceiver) has faults or the type of the needed service
changes, users need to replace the optical module. The CiTRANS 660 optical
module supports the hot insertion. You can operate on the optical module which
needs replacement during the routine maintenance as required.
Prerequisite
A spare optical module of the same type as the optical module to be replaced is
prepared.
u The OTNM2000
4 Dedicated cleaning solvent (the first choice is the isoamyl alcohol and the
second choice is the isopropyl alcohol )
Precautions
u Before the SFP / XFP optical module is replaced, check whether the optical
interface where the optical module is to be replaced has been configured the
protection.
Version: A 53
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
4 If the protection is not configured, replacing the optical module will cause
the service interruption.
u Never look directly into the end face of the optical fiber or connector when
operating the optical interface card or optical fiber.
Operation procedures
1. Put on the ESD protection wrist strap (ensure that the plug is inserted in the
ESD protection earth grounding fastener correctly).
2. Check whether the optical interface of the optical module to be replaced has
been configured the protection.
1) Mark the corresponding relationship between the optical fiber and the
optical interface and disconnect the optical fiber connecting with the optical
module.
3) Unplug the optical module, put it into the ESD protection bag, attach the
maintenance label and record the name and fault symptoms of the NE as
indicated by arrows in ② of Figure 4-3.
54 Version: A
4 Common Operations of Troubleshooting
4) Insert the standby optical module as indicated by arrows in ①of Figure 4-4.
5) Clean the optical fiber connector (See Cleaning an Optical Fiber Connector)
and plug it into the corresponding optical interface.
4. In the OTNM2000, check the related alarms and performance events of the
card where the replaced optical module is located, so as to confirm that no
abnormal alarms and performance events are caused by the new optical
module.
Version: A 55
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Prerequisites
The spare pigtail of the same type as the pigtail to be replaced is prepared.
u Dedicated cleaning solvent (the first choice is the isoamyl alcohol and the
second choice is the isopropyl alcohol )
u Adhesive tape
Operation procedures
1. Put on the ESD protection wrist strap (ensure that the plug is inserted in the
ESD protection earth grounding fastener correctly).
2. Hold the latch of the fiber connector, unplug the pigtail, and cover the fiber with
the plastic protection cover.
Note:
3. Select a spare pigtail fiber of the same type as the one to be replaced.
4. Remove the plastic protection cover of the new pigtail connector, and put it into
the protection cover box.
5. Clean the pigtail connector with the non-woven lens tissue with the cleaning
solvent.
6. Slowly insert the latches of the pigtail connector in the direction vertical to the
socket until a click is heard.
56 Version: A
4 Common Operations of Troubleshooting
Prerequisites
Precautions
Operation procedures
1. Put on the ESD protection wrist strap (ensure that the plug is inserted in the
ESD protection earth grounding fastener correctly).
2. Firmly depress the snap-in latch of the fan. For the location of the snap-in latch
of the fan unit, see Figure 4-5.
3. Draw the fan unit out steadily for a distance of 5mm so that the power is
removed from the fan.
4. Wait a minute, and then draw the fan unit out completely from the subrack, as
shown in Figure 4-6
Version: A 57
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
5. Place the fan unit that has been replaced into the ESD protection bag. Attach a
maintenance label, which records the name of the NE and the fault symptom.
6. Hold the new fan unit and align its right and left slide rails with the right and left
slide rail grooves in the fan unit area in the subrack.
7. Push the fan unit to the inside of the subrack slowly, until the snap-in latch is
secured automatically, as shown in Figure 4-7.
58 Version: A
4 Common Operations of Troubleshooting
8. Observe the indicator LED of the fan panel. The ACT indicator LED should be
solid green.
9. View the alarms of the NMUJ1 card via the OTNM2000, and confirm that no fan
alarms exist.
u A blower
Version: A 59
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Operation procedures
2. Pull the handle to drag the anti-dust screen out of the subrack, as shown in
Figure 4-8.
4. Wipe the anti-dust screen with a dry cloth and dry it using a blower.
5. Hold the cleaned anti-dust screen by two hands and align its left and right slide
rails with the left and right slide rail grooves in the anti-dust screen area of the
subrack.
6. Push the anti-dust screen slowly until it is completely inserted into the socket of
the subrack back plane, as shown in Figure 4-9.
60 Version: A
4 Common Operations of Troubleshooting
The loopback includes the hardware loopback and the software loopback.
Prerequisites
None.
Caution:
Operation procedures
u Optical interface
Connect the Tx end and Rx end of the optical interfaces using a fiber jumper to
loop back optical signals.
Version: A 61
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Connect the Tx port and Rx port of the Ethernet signal using a dedicated
Ethernet loopback head to loop back Ethernet signals.
u E1 interface
Connect the Tx port and Rx port of the E1 signal using a loopback cable to loop
back the E1 signals.
62 Version: A
4 Common Operations of Troubleshooting
Note:
Prerequisites
u The OTNM2000 can normally manage, configure, and monitor all NEs.
u The user has obtained the network management authority with an intermediate
level or above.
Caution:
The OTNM2000
Version: A 63
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Operation procedures
1. In the OTNM2000 GUI, click the NE which the card belongs to and enter the
NE view.
2. In the subrack view, right-click the NE and select "Control Command from the
shortcut menu to bring up the control command window.
4 Equipment loopback
c) In the "Loopback Objects field, select the links, slots, ports or lines to
be looped back.
d) Click the button next to the device loopback, and issue the
loopback command.
4 Line loopback
a) In the control command tab, click the button next to the line
loopback item.
c) In the "Loopback Objects field, select the links, slots, ports or lines to
be looped back.
d) Click the button next to the line loopback, and issue the loopback
command.
Note:
Clear the loopback: Select the loopback objects, and set Disable in the
Enable/Disable field to clear the loopback.
64 Version: A
4 Common Operations of Troubleshooting
Dust or dirt on the optical fiber connector end can influence the optical signal quality,
lower the system performance, and even bring silent failures that affect the reliable
operating of the equipment. Clean the connector, so as to prevent degraded
transmission performance of the equipment.
Prerequisites
It is recommended that you conduct the cleaning operations at night when the
service traffic is at a relatively low volume.
u The OTNM2000
4 Dedicated cleaning solvent (the first choice is the isoamyl alcohol and the
second choice is the isopropyl alcohol )
Version: A 65
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Note:
u To prevent laser radiation from injuring eyes, do not look into the end
face of the optical interface or fiber connector directly with naked
eyes.
Operation procedures
1. Put on the ESD protection wrist strap (ensure that the plug is inserted in the
ESD protection earth grounding fastener correctly).
2. Via the card control command GUI of the OTNM2000, issue the laser shutdown
command to the corresponding optical interface. Then unplug the optical fiber
to be cleaned.
4. Use the optical fiber connector cleaner or other cleaning tools to clean the end
of the optical fiber connector.
5. Connect the cleaned optical fiber connector with the optical interface of the
card.
6. Via the card control command GUI of the OTNM2000, issue the laser switch-on
command to the corresponding optical interface. Then unplug the optical fiber
to be cleaned.
7. Use an optical power meter to test whether the Tx / Rx optical power is in the
normal range. For the detailed parameters and specifications of the optical
power, refer to CiTRANS 660 High-Capacity MPLS-TP Platform Product
Description.
66 Version: A
4 Common Operations of Troubleshooting
Measure whether the Tx optical power of the optical interface card to be tested
meets relevant specifications, making sure that the Tx optical power of the optical
interface card is in the normal range.
Prerequisite
Operation connection
Version: A 67
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Operation procedures
1. Unplug the optical fiber of the optical interface to be tested (the TX port) and
cover the optical fiber connector with an anti-dust cap.
2. Set the wavelength of the optical power meter in accordance with the operating
wavelength of the optical interface to be tested.
68 Version: A
4 Common Operations of Troubleshooting
Caution:
To ensure normal operations of the optical power meter, please first set
the wavelength of the optical power meter in accordance with the
operating wavelength of the optical interface to be tested. Then continue
to perform the next operations.
3. Use the testing pigtail fiber that matches the interface to connect the optical
interface (the TX port) and the optical power meter.
4. Observe the reading on the optical power meter. When the reading is stable,
the optical power value at this moment is the actual mean launched power of
the optical interface to be tested.
5. Check the tested optical power according to the parameter label on the optical
module of the tested optical interface and the technical specifications in Optical
Interface Specifications.
6. If the mean launched power of the optical interface is tested beyond the normal
range, follow the steps below:
7. Remove the cap from the original fiber connector and reconnect the fiber to the
tested optical interface (TX interface).
8. On the ODF side, test the input power transmitted by the tested card and
compare it with the tested power tested at the interface on the tested card.
4 The difference is equal to or larger than 0.5dBm: Abnormal. Check the fiber.
If the fiber is faulty, e.g., the fiber is seriously bent, replace the fiber
between the tested card's optical interface and the ODF. Measure the
power again until the difference is smaller than 0.5dBm.
9. Follow Step 1 to Step 8 to test the mean launched power at all interfaces on the
equipment..
Version: A 69
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Measure whether the receiving optical power of the optical interface card meet
relevant specifications, making sure that the actual receiving optical power values of
all the optical interfaces on the equipment are in the normal range.
Prerequisite
Optical fibers at the neighboring station have been connected to the ODF of the
local station and provide optical signals to the local station.
u Optical attenuator
u Fiber flange
Operation connection
Figure 4-14 illustrates the connection diagram for testing the actual Rx optical power
at the optical interface at the local station.
1. Set the wavelength of the optical power meter so that it is consistent with the
operating wavelength of the optical interface to be tested.
70 Version: A
4 Common Operations of Troubleshooting
2. Unplug the optical fiber that is connected to the optical interface to be tested
(RX port) of the local station. Then connect the optical interface to the optical
power meter.
3. Observe the reading on the optical power meter. When the reading is stable,
the optical power value at this moment is the actual received power of the
optical interface to be tested.
Note:
The normal range of the actually received optical power is sensitivity
+3dBm<actually received power<overload point -5dBm.
1) Check whether the optical fiber connector, optical attenuator and the fiber
flange are contaminated.
¡ If the ODF side flange and the optical attenuator are contaminated,
replace the flange and the attenuator.
7. Follow Step 1 to Step 6 to test the actually received optical power at all
interfaces on the equipment.
The following introduces how to view the alarms and performance during
troubleshooting.
View the current alarms regularly to isolate and eliminate faults in a timely manner.
Maintenance cycle
Daily
The OTNM2000
Prerequisites
u The user has obtained the network management authority with an intermediate
level or above.
72 Version: A
4 Common Operations of Troubleshooting
1. Double-click the icon on the desktop and enter your username and
password to access the OTNM2000 window.
2. Click Alarm→Current Alarm View on the menu bar to bring up the operation
tree pane, as shown in Figure 4-15.
4. Click the button on the top of the object tree in the Operation Tree pane,
view the current alarms of the selected object, as shown in Figure 4-15.
1. Double-click the icon on the desktop and enter the username and
password to access the OTNM2000 window.
2. Click Alarm→History Alarm View on the menu bar to bring up the operation
tree pane, as shown in Figure 4-16.
Version: A 73
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
4. Click the button on the top of the object tree in the Operation Tree pane,
view the alarm history of the selected object, as shown in Figure 4-16.
Reference Standard
Fault handling
If any abnormal alarm exists in the system, refer to alarm and performance
reference for troubleshooting.
By querying the reported performance, users can check whether the equipment is in
steady operation status, so as to detect any silent faults as soon as possible and
deal with any abnormal situation in a timely manner. Normally the performance data
of the NMUJ1 card and the service interface card are queried.
74 Version: A
4 Common Operations of Troubleshooting
u From the NMUJ1 card's performance data, the user can detect the abnormal
temperature or power voltage of the equipment in a timely manner.
u From the service interface card's performance data, the user can obtain the
error count of the system.
Maintenance cycle
Daily
The OTNM2000
Prerequisites
u The user has obtained the network management authority with an intermediate
level or above.
Note:
1. Double-click the icon on the desktop and enter the username and
password to access the OTNM2000 window.
4 Query the NE's current performance data: Right-click the desired NE in the
Logical Tree on the left side of the OTNM2000 GUI, and select Current
Performance from the shortcut menu to bring up the current performance
display.
Version: A 75
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
4 Query the card's current performance data: Click the desired card in the
NE view, and select Current Performance on the Task Panel on the right
to bring up the current performance display, as shown in Figure 4-17.
1. Double-click the icon on the desktop and enter the username and
password to access the OTNM2000 window.
4. Click the button on the top of the object tree in the Operation Tree pane
and select History 15-Minute Performance or History 24-Hour Performance
of the selected object, as shown in Figure 4-18.
76 Version: A
4 Common Operations of Troubleshooting
Reference Standard
u The equipment has no errors, packet loss or error packets. See the relevant
performance codes and names in Table 4-3.
Performance
Card Type Performance Name Recommended Value
Code
TEMP EMU temperature 10℃ to 50℃
Network
Power supply voltage of
management POWER -41V to -57V
cabinet
signaling control
The highest temperature in
card FAN_HTEMP 10℃ to 50℃
the cooling area of the fan
Working temperature of
LASER_TEMP 0℃ to 40℃
Optical interface laser
card See Optical Interface
IOP, OOP Input / output optical power
Specifications.
Version: A 77
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Table 4-3 Performance codes of card error, packet loss and error packet count
Fault handling
For further information about abnormal performance parameters, refer to Alarm and
Performance Reference.
78 Version: A
5 Handling Network Management
Communication Faults
Network management communication faults do not influence services, but they can
preclude centralized control of the networked NEs from the OTNM2000. So users
must handle these faults promptly. The following introduces the NE communication
faults and how to handle them.
All NEs in a Certain Network Block or the Entire Network Out of Management
Version: A 79
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
The possible causes for the fault of a single NE out of management are described
as follows:
u The clock and cross-connect card or the OTNM2000 signaling control card is
faulty.
1) Check whether the NE’s power supply is faulty: If so, troubleshoot the
power supply faults.
80 Version: A
5 Handling Network Management Communication Faults
2) Check whether the NE’s line fiber is interrupted: If so, troubleshoot the line
faults.
2. Eliminate faults of the ONTM2000 signaling control card and the IP settings.
1) Ping the NE via the OTNM2000 to read the NE status. If ping fails or the
NE status cannot be retrieved, unplug and replug the NMUJ1 card. If the
problem persists, replace the NMUJ1 card.
3. Eliminate faults of the clock and cross-connect card: View whether the clock
and cross-connect card generates abnormal alarms. If so, first handle the
alarms of the clock and cross-connect card.
4. Eliminate faults of the line card: View whether the line card (it is used to
transport the MCC information) generates abnormal alarms. If so, first handle
the alarms of the line card.
Version: A 81
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
The common causes for the fault of all NEs in a certain network block or the entire
network out of management are described as follows:
u The network cable between the directly-connected NE and the HUB is broken
or the port on the HUB is faulty.
Note:
2) Check whether the gateway NE’s line fiber in each direction is broken: If so,
troubleshoot the fiber interruption.
2. Eliminate faults of the NMUJ1 card and the IP settings of the gateway NE.
1) Ping the NE via the OTNM2000 to read the NE status. If ping fails or the
NE status cannot be retrieved, unplug and replug the NMUJ1 card. If the
problem persists, replace the NMUJ1 card.
2) Check the gateway NE’s IP setting: If it is incorrect, set the gateway NE’s
IP correctly according to the IP address assignment table (one of the
project design documents).
82 Version: A
5 Handling Network Management Communication Faults
1) Check the network cable connection between the gateway NE and the
HUB: If the connection becomes loose, secure the network cable.
¡ If all network cables are normal, replace the HUB and check whether
the fault disappears.
4. Eliminate faults of the clock and cross-connect card of the gateway NE: View
whether the clock and cross-connect card generates abnormal alarms. If so,
first handle the alarms of the clock and cross-connect card.
5. Eliminate the OTNM2000 computer’s IP setting faults: You should pay attention
to the TCP / IP properties of each NIC and the configuration of the OTNM2000
management program on the OTNM2000 host. For the basic configuration
rules, refer to e-Fim OTNM2000 Element Management System Operation
Guide.
Version: A 83
6 Handling Errors and Packet Loss
To handle errors and packet loss, you need to know the causes of these faults, and
handle them following proper steps. The following introduces information related to
errors and packet loss, and how to handle them.
Background Information
Cause of Fault
Troubleshooting Procedure
84 Version: A
6 Handling Errors and Packet Loss
When only a few errors and lost packets exist in the network, the affected services
are kept in a limited range. But the maintenance staff should isolate and eliminate
the errors and packet loss promptly, so as to prevent the faults from accumulating
and leading to service interruption.
4 The equipment room does not provide stable input voltage for the
equipment .
4 The contact between the pigtail fiber / cable and the equipment is poor.
4 The contact between the service interface card / clock and cross-connect
card / line card and the subrack backplane is poor.
u Equipment faults:
4 The service interface card, clock and cross-connect card, or line card has
faults.
Version: A 85
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
1. Analyze the fault phenomena: Ascertain the possible causes of the fault
according to the fault conditions.
Multiple factors can cause error and packet loss. So the maintenance staff
should use the related knowledge to perform ascertaining and find out the
regular pattern of the error and packet loss, so as to handle the fault correctly
and rapidly. For example:
4 If only a certain channel at the client side generates errors and loses
packet, suspect a fault in the service card / clock and cross-connect card
at the client side or the previous clock and cross-connect card.
4 If the services related to a line card inside a certain NE all has faults,
suspect a fault in this card, the optical module of this card, or the line fiber
connected with it.
2. Isolate and eliminate faults correctly: Isolate the fault points with the generally
used fault isolation methods.
4 Query alarms and performance: Ascertain the lines generating errors and
losing packets rapidly via using the alarm and performance query functions
of the OTNM2000.
Caution:
3. Record the phenomena of this fault event and the corresponding handling
procedure.
86 Version: A
7 Handling Abnormal Optical Power
Faults
Background Information
Cause of Fault
Troubleshooting Procedure
Version: A 87
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
During the start-up and daily maintenance of the CiTRANS 660, to isolate the
optical power abnormal points in the system quickly and correctly, you should be
familiar with the specifications on the input and output optical power of various types
of optical modules.
u Overlow optical power: It can cause errors and packet loss of the services, and
can even lead to complete service interruption.
4 It can cause errors and packet loss of the services, and can even lead to
complete service interruption.
Note:
Category Cause
Line performance is degraded.
88 Version: A
7 Handling Abnormal Optical Power Faults
1. Check the external faults: Check whether the Tx optical power of the client side
equipment is normal.
4 Normal: The fiber between the optical interface card and client side
equipment is faulty or the connection is incorrect. Check the pigtail, fiber
connector and line attenuation of the equipment in sequence and exclude
the faults.
4 Abnormal: Contact the client and troubleshoot the client side equipment.
2. Check the equipment: Check whether the optical module of the local
equipment.
4 Abnormal: Replace the SFP or the XFP optical module of the card.
Version: A 89
8 Handling L2VPN Service Interruption
In terms of service type, the L2VPN service interruption can be classified into CES
interruption and Ethernet service interruption. To handle the faults, you need to
know the causes of these faults, and then handle them following proper steps. The
following introduces L2VPN service interruption and how to handle it.
90 Version: A
8 Handling L2VPN Service Interruption
The following introduces CES service interruption and how to handle it.
u Card failure;
u Incorrect configuration.
Category Cause
Card hardware failure or the inter-card communication failure.
Signal loss or degradation received at the interface card or the
CES service interruption processing card.
Version: A 91
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
92 Version: A
8 Handling L2VPN Service Interruption
4 Loopback: If possible, narrow down the search scope and isolate the
specific station where the CES failure occurs using the loopback method.
After isolating the faulty station, refer to Cause of Fault to exclude suspected
faulty points. Follow the sequence to check and analyze:
5) Check whether alarms of clock failure such as AU_LOP and LTI occur.
Version: A 93
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
The following introduces Ethernet service interruption and how to handle it.
The CiTRANS 660 cards with FE, GE or XGE interfaces all support Ethernet
services. The common Ethernet service failures include
Category Cause
Card failure or the inter-card communication failure.
Communication loss or degradation.
Card failure.
An abnormal optical power failure.
A rise in packet loss rate and a
Unclean fiber connector or incorrect connector.
decrease of communication
Optical cable or pigtail failure.
quality of the Ethernet service.
Abnormal card temperature.
94 Version: A
8 Handling L2VPN Service Interruption
Version: A 95
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
4 Loopback: If possible, narrow down the search scope and isolate the
specific station where the Ethernet service failure occurs using the
loopback method.
After isolating the faulty station, refer to Cause of Fault to exclude suspected
faulty points. Follow the sequence to check and analyze:
3) Check whether the statuses of the Ethernet interface and the interface at
the opposite equipment are consistent.
¡ Check the flow control and MTU values set at both ends of the line.
7) Check the link service VLAN, VPWS, VPLS and flow configuration.
96 Version: A
8 Handling L2VPN Service Interruption
¡ Check the line service VLAN, VPWS, VPLS and flow configuration
and redeliver the cross-connect configuration.
Version: A 97
9 Handling QoS Failure
Background Information
Cause of Fault
Troubleshooting Flow
Troubleshooting Procedure
98 Version: A
9 Handling QoS Failure
Category Cause
Version: A 99
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Alarm and performance analysis: Analyze the network alarms and performance
data according to failed service information to isolate the possible station where
the QoS abnormality occurs.
After isolating the faulty station, refer to Cause of Fault to exclude suspected
faulty points. Follow the sequence to check and analyze:
100 Version: A
9 Handling QoS Failure
Version: A 101
10 Handling Network Level Protection
Switching Faults
Users should handle network level protection switching faults promptly. If these
faults are not handled in a real-time manner, the protection switching of key services
may fail, and even lead to service interruption. The following introduces the network
level protection switching faults and how to handle them.
Background Information
Cause of Fault
Troubleshooting Flow
Troubleshooting Procedure
102 Version: A
10 Handling Network Level Protection Switching Faults
u The entire network does not enter the protection switching status or the
protection switching status is incorrect when the protection switching is needed.
u Switching is conducted without any reason when the entire network is working
normally.
u In the protection switching status, the services of the entire or partial network
are interrupted.
The common causes of the network level protection switching fault are listed in
Table 10-1.
Table 10-1 Common causes of the network level protection switching fault
Category Cause
The optical fibers are not correctly connected.
The troubleshooting process for a network level protection switching fault is shown
in Figure 10-1.
Version: A 103
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Figure 10-1 Troubleshooting flow of the network level protection switching failure
104 Version: A
10 Handling Network Level Protection Switching Faults
1. Isolate the line and the station where the network level protection switching
occurs.
Alarm and performance analysis: Analyze the network alarms and performance
data according to failed service information to isolate the possible station where
the network level protection switching failure occurs.
After isolating the faulty station, refer to Cause of Fault to exclude suspected
faulty points. Follow the sequence to check and analyze:
¡ Query the current status of each NE via the OTNM2000: If the SWR
alarm occurs at multiple adjacent NEs in the network, the switching
protocol processing is normal. Follow the ordinary service interruption
handling procedure to check whether the card is faulty.
Version: A 105
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
¡ Exclude the protection line fault: Query whether the fiber line alarms
such as IOP_HIGH/IOP_HIGH, IOP_LOW/IOP_LOW, LASER_CCT
and RLOS exist at the protection related NE interface cards.
106 Version: A
11 Fault Case
Version: A 107
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
See Figure 11-1 for the network topology of this example. NE1, NE2 and NE3 form a
ring using the GSJ2 cards in Slot 06 and Slot 0D. NE1 and NE3 are connected to
the client equipment respectively. Several FE services exist between NE1 and NE3
and the service route is NE1→NE2→NE3. Some of the FE services from NE1 to
NE3 are interrupted.
Fault analysis
From the fault symptom, analyze whether the optical line (VS layer) is faulty. If so,
the services carried by the optical line are affected. In this example, the fault
symptom is described as some FE services are interrupted. The fault point is
probably not at the optical layer (VS layer). The fault causes may include
u The line between the client side equipment and the transmission equipment is
faulty.
u The CiTRANS 660's clock and cross-connect card of the equipment is faulty.
108 Version: A
11 Fault Case
Troubleshooting procedure
1. Exclude the client side equipment and line faults: Perform tests with meters on
the corresponding port of the faulty service. The result shows the Rx and Tx
data are abnormal. The client side equipment and line faults are excluded.
Version: A 109
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Figure 11-2 Example of tributary card failure - VP layer loopback frame configuration
110 Version: A
11 Fault Case
Figure 11-3 Example of tributary card failure - VP layer CV frame configuration of NE1
1) Access the clock and cross-connect configuration panes of NE1 and NE3.
In the PW tab, check corresponding VC layer configuration of the faulty
LSP. The NE3's related information is consistent with the NE1's.
Version: A 111
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Figure 11-4 Example of tributary card failure - VC layer loopback frame configuration
112 Version: A
11 Fault Case
4) Repeat the previous steps to test the VC services on the faulty LSP. Isolate
the fault to the specific card.
4. Now the fault scope can be narrowed down to an FE line card or a clock and
cross-connect card.
1) Replace the faulty card of the CiTRANS 660 using the substitution method.
Version: A 113
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Note:
Summary
The CiTRANS 660 has a comprehensive OAM mechanism to detect service faults
at all layers and narrow down the fault scope to a specific card.
The example in Some Services Interrupted due to Tributary Card Failure is applied
here and the network topology is shown in Figure 11-6. NE1, NE2 and NE3 form a
ring using the GSJ2 cards in Slot 06 and Slot 0D. NE1 and NE3 are connected to
the client equipment respectively. Several FE services exist between NE1 and NE3
and the service route is NE1→NE2→NE3. The FE services from NE1 to NE3 are all
interrupted.
114 Version: A
11 Fault Case
Fault analysis
In this example, the fault symptom is described as FE services are all interrupted.
The fault point is probably at the optical layer (VS layer). The fault causes may
include
u The line between the client side equipment and the transmission equipment is
faulty.
u The CiTRANS 660 clock and cross-connect card of the equipment is faulty.
Troubleshooting procedure
1. Exclude the client side equipment and line faults: Perform tests with meters on
the corresponding port of the faulty service. The result shows the Rx and Tx
data are abnormal. The client side equipment and line faults are excluded.
Version: A 115
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
1) Exclude the alarm information of all NE line cards that the faulty service
passes: Check whether the optical line alarms such as IOP_HIGH,
IOP_LOW, LASER_TF and LINK_LOS occur.
2) Test the line cables to see whether any cable has overhigh attenuation.
Exclude the line faults.
3) Exclude optical line faults using VC layer OAM frame. In the Demanded-
OAM-Config tab of the GSJ2 card configuration pane of NE1 and NE4,
add an OAM item, as shown in Figure 11-7.
4) Right-click the clock and cross-connect cards of NE1 and NE3, select
Status Monitor from the shortcut menu to query the VS layer OAM frame
performance. If the Rx LBR frame quantity = Tx LBM frame quantity, the
VS link channel is normal; otherwise, the link channel is faulty.
5) From the above analysis, isolate the faulty optical section. Find the faulty
line card using the substitution method. After replacing the line card, all FE
services have recovered.
116 Version: A
11 Fault Case
See Figure 11-8 for the network topology of this example. NE1 and NE2, NE3 and
NE4 are connected using the GSK1 cards in Slot 11 and Slot 10. NE1 and NE3
(using the GSK1 cards in Slot 11) are connected with NE5 (using the GSJ2 cards in
Slot 05 and Slot 06). NE5 (using the GSJ2 cards in Slot 08 and Slot 0D) is
connected with the client RNC equipment. The fiber in Slot 08 is used to carry
service of NE1 and NE2, and the fiber in Slot 0D is used to carry service of NE3 and
NE4.
The service of NE1 and NE2 is normal, whereas the service of NE3 and NE4 is
interrupted. No alarms are reported to the OTNM2000.
Fault analysis
From external to internal analysis, the fault may be caused by the following factors:
Version: A 117
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
Troubleshooting procedure
1) Ping the base station equipment connected to the NEs from the RNC
equipment to check the network connectivity. The base station equipment
connected to NE1 and NE2 can be pinged, whereas the base station
equipment connected to NE3 and NE4 cannot.
2) Ping each sub-NE from NE5 and find that the sub-NEs can be pinged.
3. From above operations, suspect a problem in the NE3 and NE5 equipment, or
in the connection between NE5 and the RNC equipment.
1) Check the OTNM2000 configuration of NE3 and NE5 and find that no
abnormal alarms or performance appears, but the Rx power performance
is slightly low.
2) Test the optical fiber between NE5 and the RNC equipment using an
optical power meter. The result shows the fiber Rx power in Slot 08 is
within the card's normal range, whereas the Rx power in Slot 0D is beyond
the normal range.
Check the line and find that a 10GB optical attenuator is installed on the fiber in
Slot 0D connecting with the RNC equipment. Remove the attenuator and
reconnect the fiber to eliminate the fault.
Summary
Based on experience in laying out fibers for the TDM equipment in the central office,
an optical attenuator was added to the fiber in Slot 0D to avoid overhigh optical
power. However, this operation led to overlow Rx power of the card and further
caused service interruption.
118 Version: A
11 Fault Case
Attenuators should be added only when the actually tested Rx power is beyond the
normal range of the optical receiver of the equipment (refer to Optical Interface
Specifications). Do not make subjective assumptions during installation and
construction.
Version: A 119
Appendix A Indicator LEDs
On the door lintel of the cabinet are three indicator LEDs in different colors. See
Table A-1 for the descriptions of the three indicator LEDs.
The indicator LEDs on the fan unit are used to indicate the operating status of the
fan unit. See Table A-2 for descriptions of the indicator LEDs on the fan unit.
120 Version: A
Appendix A Indicator LEDs
The indicator LEDs on the card panels are used to indicate the operation, alarms
and other statuses of the cards.
Version: A 121
CiTRANS 660 High-capacity MPLS-TP Platform Troubleshooting Guide
MCC
Blinking green: the management plane data exists
MCC communication
on the optical line.
indicator LED
SCC
Blinking green: the control plane data exists on the
SCC communication
optical line.
indicator LED
122 Version: A
Appendix A Indicator LEDs
Version: A 123
Appendix B Optical Interface
Specifications
In the CiTRANS 660, the XSJ2, XSJ3 and XGJ1 cards use 10GE optical interface
modules. See Table B-1 for the parameter specifications.
Item Specification
In the CiTRANS 660, the GSJ2, GSJ3 and XGJ1 cards use GE optical interface
modules. See Table B-2 for the parameter specifications.
Item Specification
1000BASE-
Optical interface type 1000BASE-SX 1000BASE-LX 1000BASE-VX
ZX1
Source type MLM MLM SLM SLM
Target distance (km) 0.55 10 40 80
Mean launched power
-9.5 to 0 -8 to -3 -2 to 3 -2 to 5
(dBm)
Central wavelength
770 to 860 1270 to 1355 1275 to 1350 1500 to 1580
(nm)
Overload optical
0 -3 -3 -3
power (dBm)
124 Version: A
Appendix B Optical Interface Specifications
Item Specification
Receiving sensitivity
-17 -20 -23 -23
(dBm)
Minimum extinction
9 9 9 9
ratio (dB)
In the CiTRANS 660, the ESJ2 card use the FE optical interface module. See
Table B-3 for the parameter specifications.
Item Specification
In the CiTRANS 660, the S1J1, S1J2, S1O1 and S1J4 cards use the STM-1 optical
interface modules. See Table B-4 for parameter specifications.
Item Specification
Version: A 125
Appendix C Abbreviations
IP Internet Protocol
IC Integrated Circuit
126 Version: A
Product Documentation Customer Satisfaction Survey
Thank you for reading and using the product documentation provided by FiberHome. Please take a moment to
complete this survey. Your answers will help us to improve the documentation and better suit your needs. Your
responses will be confidential and given serious consideration. The personal information requested is used for
no other purposes than to respond to your feedback.
Name
Phone Number
Email Address
Company
To help us better understand your needs, please focus your answers on a single documentation or a complete
documentation set.
Documentation Name
Code and Version
12. Additional comments about our documentation or suggestions on how we can improve:
Thank you for your assistance. Please fax or send the completed survey to us at the contact information
included in the documentation. If you have any questions or concerns about this survey please email at
[email protected]