Network Troubleshooting

Troubleshooting Methodologies and Tools

Objectives
• Troubleshooting networks is more important than
ever. As time goes on, services continue to be added
to networks. With each added service comes more
variables. This adds to the complexity of the network
troubleshooting as well as the network itself.
• Troubleshooting begins by looking at a methodology
that breaks down the process of troubleshooting into
manageable pieces. This permits a systematic
approach, minimizes confusion, and cuts down on
time otherwise wasted with trial and error
troubleshooting.
• One technician following a logical sequence will
almost always be more successful than a gang of
technicians, each with their own theories and
methods for troubleshooting.
 START

Collect all available information and analyze the

symptoms of failure
Localize the problem to within a single network
segment, to a single complete module or unit, or to
a single user
Isolate the trouble to a specific hardware or
software, within a unit, module, or user’s network
account
Locate and correct the specific
problem
Verify that the problem has been
solved
YES
Other problems?
NO

STOP
• Use a logical, methodical plan for determining
what needs to be fixed on the network.
• Work from Layer 1 (Physical) and climb the
OSI model layer-by-layer until you reach the
resolution…or Layer 7 (Application)…
whichever comes first.
OSI Model

Layer 7 Application Telnet, Browsers, FTP, Email

Layer 6 Presentation Encryption, Compression, Data

representation & Data syntax
Layer 5 Session Conversations & dialogs, Managing
sessions between applications
Layer 4 Transport TCP & UDP, Sliding windows,
segments, 3-way handshake, Error
checking
Layer 3 Network Routers, Path determination,
Protocols, Logical addressing
Layer 2 Data Link LLC & MAC, MAC addresses, Bridges
& switches, error-checking, Network
technologies, Encapsulation
Layer 1 Physical Bits, Cables, Connectors, hubs &
repeaters
 START

NO
Layer 1 OK? Fix Layer 1
YES

Layer 2 OK?NO Fix Layer 2

YES

OSI Layer Layer 3 OK?NO Fix Layer 3

Troubleshooti YES

NO
Layer 4 OK?
ng YES
Fix Layer 4

Layer 5 OK?NO Fix Layer 5

YES

Layer 6 OK?NO Fix Layer 6

YES

Layer 7 OK?NO Fix Layer 7

YES

STOP
OSI model versus TCP/IP model
Position of network devices in layered model
• The ability to identify which layers pertain to a
networking device gives a troubleshooter the ability
to minimize the complexity of a problem by dividing
the problem into manageable parts.
– For instance, knowing that Layer 3 issues are of no
importance to a switch, aside from multilayer switches,
defines the boundaries of a task to Layer 1 and Layer 2.
– However, it is still important to note that there are network
applications that are part of these devices that move into
Layers 4-7.
Layer 1 problems
can include:
• Broken cables
• Disconnected cables
• Cables connected to the wrong ports
• Intermittent failures in cables
• Poorly terminated cables
• Wrong cables used for wrong task
• Transceiver problems
• DCE problem
• DTE problem
• Power issues/turned off or not plugged in
Layer 2 problems
can include:
• Serial interface configuration
mistakes
• Ethernet interface configuration
mistakes
• Wrong clock rate settings on serial
interfaces
• Wrong encapsulation on serial
interfaces
– HDLC is default
• Bad NIC
Layer 3 problems
can include:
• Routing protocol not enabled
• Wrong routing protocol enabled
• Incorrect network/IP address
• Incorrect subnet mask
• Incorrect interface addresses
• Incorrect DNS to IP bindings (host table
entries)
• Wrong autonomous system number for
IGRP
TROUBLESHOOTING APPROACHES
Troubleshooting Process

Step 1 Identify the problem

Step 2 Establish a theory of probable causes
Step 3 Determine an exact cause
Step 4 Implement a solution
Step 5 Verify solution and full system functionality
Step 6 Document findings
Step 1 - Identify the Problem

• Hardware/Software information
• Manufacturer, model, OS, network environment, connection type
• Open-ended questions
• When did the problem start?
• What problems are you experiencing?
• Is there anything else you can tell me about the problem?
• What other users are having problems?
• Can you describe your network configuration?
• Closed-ended questions
• Has any network equipment changed?
• Have any peripherals been added to your computer?
• Have any other computers been added to the network?
• Have you rebooted your computer?
Step 2 - Establish a Theory of Probable Causes

• Problem may be simpler than the customer

thinks.

• Create a list of the most common reasons why the

error would occur:
• Incorrect IP information
• Examine the network equipment LEDs
• Incorrect wireless configuration
• Disable network connection
• Verify the wireless router configuration
• Verify the network equipment settings
Step 3 - Determine the Exact Cause

• Testing your theories of probable causes one at a time, starting with the
quickest and easiest.
• Restart the network equipment.
• Examine the network equipment LEDs.
• Renew the IP address.
• Reconnect all of the network cables.
• Verify the wireless router configuration.
• Ping the local host.
• Ping the default gateway.
• Ping an external website.
• Verify the network equipment settings.
• If the exact cause of the problem has not been determined after you have
tested all your theories, establish a new theory of probable causes and test
it.
Step 4 - Implement a Solution

• Sometimes quick procedures can determine

the exact cause of the problem or even
correct the problem.
• If a quick procedure does not correct the
problem, you might need to research the
problem further to establish the exact cause.
• Divide larger problems into smaller problems
that can be analyzed and solved individually.
Step 5 - Verify Solution and System Functionality

• Verifying full system functionality and implementing any preventive

measures if needed. Ensures that you have not created another problem
while repairing the computer.
• Reboot all of the network equipment.
• Reboot any computer that experienced network problems.
• Validate all LEDs on the network equipment.
• Use the ipconfig/all command to display IP addressing information
for all network adapters.
• Use the ping command to check network connectivity to an
external website.
• Use the nslookup command to query your DNS server.
• Use the net view command to show the available shared resources
on a network.
• Print to both a shared and a local printer.
• Have the customer verify the solution and system functionality.
Step 6 - Document Findings

 Discuss the solution with the customer

 Have the customer confirm that the problem has been solved
 Document the process
Problem description
Solution
Components used
Amount of time spent in solving the problem
Bottom-up
• When applying a bottom-up
approach towards
troubleshooting a networking
problem, the examination
starts with the physical
components of the network
and then is worked up
through the layers of the OSI
model until the cause of the
problem is identified.
– It is a good approach for a
troubleshooter to use when
the problem is suspected to
be physical.
– Most networking problems
reside at the lower levels, so
implementing the bottom-up
approach will often result in
Top-down
• When applying a top-down
approach towards
troubleshooting a networking
problem, the end user
application is examined first.
Then work down from the
upper-layers of the OSI
model until the cause of the
problem has been identified.
– When a troubleshooter
selects this approach, the
applications of an end
system are tested before
tackling the more specific
networking pieces.
– A troubleshooter would most
likely select this approach
for simpler problems or
when the troubleshooter
thinks that the problem is
with a piece of software
• The disadvantage to
selecting this approach is
Divide and conquer
• When the divide and conquer
approach is applied towards
troubleshooting a networking
problem, a layer is selected
and tested in both directions
from the starting layer.
• The divide and conquer
approach is initiated at a
particular layer.
• The layer is based on
troubleshooter experience
level and the symptoms
gathered about the problem.
• Once the direction of the
problem is identified,
troubleshooting follows that
direction until the cause of
Guidelines for selecting an approach
Guidelines for selecting an approach
– Determine the scope of the problem
A troubleshooting approach is often selected
based on its complexity. A bottom-up approach
typically works better for complex problems. Using
a bottom-up approach for a simple problem may
be overkill and inefficient. Typically, if symptoms
come from users then a top-down approach is
used. If symptoms come from the network, a
bottom-up approach will likely be more effective.
– Apply previous experiences
If a particular problem has been experienced
previously, then the troubleshooter may know of a
way to shorten the troubleshooting process.
– Analyze the symptoms
The more known about a problem, the better the
chance that it can be solved. It may be possible to
Guidelines for selecting an approach

• Example
– Two IP routers have been identified in a network
that have connectivity, but are not exchanging
routing information. Before attempting to solve
the problem, a troubleshooting approach needs to
be selected. Similar symptoms have been seen
previously, which point to a likely protocol issue.
Since there is connectivity between the routers, it
is not likely to be a problem at the physical or
data link layer. Based on this past experience
knowledge, it is decided to use the divide and
conquer approach, and the troubleshooter begins
testing the TCP/IP-related functions at the network
layer.
GATHERING SYMPTOMS
Gathering symptoms for a network problem
• Stage 1
The troubleshooter analyzes symptoms gathered from the trouble
ticket, users, or end systems affected by the problem to form a
definition of the problem.
• Stage 2
If the problem is in the troubleshooter’s system, it will be necessary to
move on to stage 3. If the problem is outside the boundary of the
troubleshooter’s control, it will be necessary to contact an
administrator for the external system before gathering additional
network symptoms.
• Stage 3
The troubleshooter determines if the problem is at the core,
distribution or access layer of the network. At the identified layer use
an analysis of existing symptoms and knowledge of the network
topology to determine which piece or pieces of equipment are the most
likely cause.
• Stage 4
Using a layered troubleshooting approach, the troubleshooter gathers
hardware and software symptoms from the suspect devices. The
technician starts with the most likely possibility and uses knowledge
and experience to determine if the problem is more likely a hardware
or software configuration problem.
• Stage 5
Document any hardware or software symptoms. If the problem can be
Gathering symptoms for a network problem
Gathering symptoms from an end-user: hardware
• Physical symptoms may be
related, but not limited, to
the following:
– Electromagnetic
Interference (EMI) from radio
and television transmitters,
or the introduction of
portable devices that create
EMI to the area
– Indicator lights of a NIC or
networking device
– Cable connections, the
crimping of connectors and
the physical state of
connection sockets
– Incorrect seating of modules
and cards
– Burning smells from
insulative material which
has melted, or of burnt out
components
Gathering symptoms from an end-user: software
• The troubleshooter should
use effective questioning
techniques to document the
symptoms of a problem:
– Ask questions that are
pertinent to the problem.
– Use each question as a
means to either eliminate or
discover possible problems.
– Speak at a technical level
that the user can
understand.
– Ask the user when the
problem was first noticed.
– Ask the user to re-create the
problem, if possible.
– Determine the sequence of
events that took place
before the problem
happened.
– Match the symptoms that
the user describes with
common problem causes.
Questions to ask an end-user
• A typical format for
interviewing an end user
concerning their problem is:
– What does not work?
– What does work?
– Are the things that do and
do not work related?
– Has the thing that does not
work ever worked?
– When the problem was first
noticed?
– What has changed since the
last time it did work?
– Did anything unusual
happen since the last time it
worked?
– When exactly does the
problem occur?
– Can the problem be
reproduced and if so, how
can it be reproduced?
Flow charts for gathering network and end-user
symptoms
• Stage 1 Interview user – If
possible, a troubleshooter gathers
initial symptoms from the user and
uses these symptoms as a basis for
additional troubleshooting.
• Stage 2 Analyze symptoms – A
troubleshooter will get a description
of the problem by analyzing any
gathered symptoms from the user
• Stage 3 Determine symptoms –
Using a layered troubleshooting
approach, a troubleshooter gathers
hardware and software symptoms
from the end system starting with
the most likely cause. The
troubleshooter should rely on
previous experience, if possible, to
decide if the problem is more likely a
hardware or software problem.
• Stage 4 Document symptoms –
Document any hardware and
software symptoms. If the problem
can be solved using the documented
symptoms, a troubleshooter solves
the problem and documents the
solution. If the problem cannot be
solved at this point, then the
isolating phase of the general
NETWORK MANAGEMENT TOOLS
Network management system frameworks
• Network management can range from a solitary
network consultant monitoring network activity with
a simple protocol analyzer, to the use of a distributed
database with auto-polling of network devices.
• Network management could even include the use of
high end workstations generating real time graphical
views of network topology changes and traffic.
• In general, network management is a service that
employs a variety of tools, applications, and devices
to assist network managers in monitoring and
maintaining networks.
– End stations, or managed devices, such as computer
systems and other network devices, run software that
enables them to send alerts when they recognize problems.
– Upon receiving these alerts, management entities are
programmed to react by executing one action, or a group of
actions.
– Management entities can also poll end stations to check the
values of certain variables.
Network management system frameworks
ISO Network Management Model
• Performance management – The goal is to measure and provide
various aspects of network performance in order to maintain
internetwork performance at an acceptable level. Examples of
performance variables include network throughput, user response
times, and line utilization.
• Configuration management – The goal is to monitor network and
system configuration information so that the effects on network
operation of various versions of hardware and software elements can
be tracked and managed.
• Accounting management – The goal is to measure network-
utilization parameters so that individual or group use of the network
can be regulated appropriately. Such regulation minimizes network
problems because network resources can be apportioned based on
resource capacities, and maximizes the fairness of network access
among users.
• Fault management – The goal is to detect, log, notify users of, and if
possible, automatically fix network problems to keep the network
running effectively. Because faults can cause downtime or
unacceptable network degradation, fault management is perhaps the
most widely implemented of the ISO network management elements.
• Security management – The goal is to control access to network
resources to prevent the network from being sabotaged or
unintentionally brought down, and to prevent sensitive information
from being accessed by unauthorized users. A security management
Knowledge base tools
• Knowledge bases represent an invaluable tool for the
network troubleshooter.
– The most visible example of a knowledge base is the
Internet.
• A knowledge base is often a database collection of
empirical information on a specific technical area.
– It consists of real solutions to problems encountered on a
single or on numerous occasions.
– The art in knowledge base tools is in the identification of
keywords or phrases that are associated with particular
problems and problem areas.
• The Cisco Systems website reached at
http://www.cisco.com/ incorporates a free knowledge
base tool on Cisco related hardware and software.
Knowledge base tools
Performance measurement and reporting tools
Event and fault management tools
• Pair testers – This category of
tester is distinguished from a
continuity tester by its ability to
detect a split pair. A split pair is the
simplest problem related to
frequency that plagues network
cable, and should be a minimum
entry point for a network cable
tester. If a cable passes the split pair
test, it has a good chance of passing
a basic cablecertification test.
• Frequency based field
certification testers (cable
analyzers) – The first generation of
field certification testers usually
offered selections for Category 5
cable types with a maximum
frequency rating of 10 or 20 MHz.
The second generation of field
certification testers became
available in late 1995, and offered
100 MHz testing. This generation of
tester is capable of certifying
Category 5 cable, but not Category
5e cable (both 100 MHz). The third
generation of field certification
testers became available in late
1997, and offered testing to
Policy management tools
• The Need for Policy-Based Security
Complex security technologies are necessary to
protect highly available mission critical networks
from corruption and intrusion.
– Of particular interest in the past few years is protecting
geographically dispersed enterprise networks, which use a
combination of public and private WAN lines to connect
remote and branch offices to major centers.
– Intranets, extranets, Internet connections, WANs, and LANs
each have unique security requirements. Many companies
wish to extend their mission-critical applications to remote
offices by way of an intranet, or communicate directly with
industry partners, suppliers, and key customers through
extranets.
– These technologies enable organizations to securely conduct
business in today's open environments.
Policy management tools

• Scaling Networks and Maintaining

Security
In very large networks, scalability issues can
make security deployment quite expensive
and can lead to misconfigured systems and
inconsistent policy enforcement. No
centralized, coordinated mechanism exists to
implement a consistent policy throughout the
network, verify that it is installed and
functioning properly, change it easily as
required, or detect attacks, mistakes, and
misuse within the network.
Summary
Summary
• The reader should be able to list the stages of
encapsulated data flow process. Also the reader
should be able to compare the logical layers of the
OSI and TCP/IP networking models and identify the
logical layers used by devices on a network.
• The reader should understand:
– The stages of the general troubleshooting process
– The bottom-up troubleshooting approach
– The top-down troubleshooting approach
– The divide and conquer troubleshooting approach
– How to select an effective troubleshooting approach based
on a specific situation
– The process of gathering symptoms from a network
– Guidelines for gathering symptoms from a user
– The process of gathering symptoms from an end-system