Troubleshooting

LAN Protocols

BRKRST-3131

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved. 1

Presentation_ID.scr
 Agenda

ƒ Session Overview
ƒ Troubleshooting Layer 1, Layer 2, and Layer 3
Connectivity Issues
ƒ Spanning Tree Protocol
ƒ Security
ƒ Common Issues for High CPU Utilization

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

Session Overview

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved. 2

Presentation_ID.scr
 Related Sessions

ƒ RST-3141: Troubleshooting Cisco Catalyst 3750, 3550,

and 2900 Series Switches by Michel Peters
Tuesday 2:00 PM, Wednesday 4:30 PM, Thursday 4:30 PM

ƒ RST-3142: Troubleshooting Cisco 4500 Series

Switches by Wendy Hower
Tuesday 4:30 PM, Thursday 10:30 AM

ƒ RST-3143 Troubleshooting Catalyst 6500 Series

Switches by Barnaby Dianni
Wednesday 2:00 PM, Thursday 2:00 PM, Thursday 4:30 PM

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

Networking Concepts and Operations

ƒ Be familiar with switching and routing concepts

ƒ Understand the configurations on network devices
ƒ Know what features are active and where
ƒ Be familiar with Cisco’s web sites
Configuration guides
Release notes
Troubleshooting tips
Software download page

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved. 3

Presentation_ID.scr
 Building Codes Reduce
the Severity of Disasters

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

Network Diagram

Corporate Network
Gig 0/1 Gig 0/1
155.3.77.254 155.3.76.254
VLAN187 VLAN186
HSRP Active of
Odd Vlans 1/1 Root: 1,3,5,7,9,187 1/1 Root: 2,4,6,8,186
0001.c912.7800 0003.6b73.9700 HSRP Active
DIST-01 F: 1,2,3,4,5,6,7,8,9 DIST-02 for Even Vlans
192.168.1.1 3/1–4 3/1–4 192.168.1.2

F: 1,2,3,4,5,6,7,8,9
4/4
F: 1,8,9 4/1
4/3 F: 1,2,3
4/1 4/2 F: 1,6,7 4/2 4/3 4/4
F: 1,2,3 F: 1,4,5 F: 1,4,5 F: 1,6,7 F: 1,8,9

B: 2 B: 4 B: 1,5 B: 6 B: 1,7 B: 8 B: 1,9

1/1 1/2 1/1 1/2 1/1 1/2 1/1 1/2
B: 1,3

0001.c967.7800 0001.c9dd.7800 0001.c932.9700 0001.c949.9700

Closet1 Closet2 Closet3 Closet4
192.168.1.11 192.168.1.12 192.168.1.13 192.168.1.14

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved. 4

Presentation_ID.scr
 Have a Plan

ƒ Don’t assume anything

ƒ Define the problem
ƒ Understand what is working and what is not
ƒ Is it intra-VLAN or inter-VLAN issue?
ƒ Perform basic troubleshooting
ƒ Keep the network diagram handy
ƒ Keep a protocol analyzer handy
ƒ Keep modem access ready for TAC support

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9

Agenda

ƒ Session Overview
ƒ Troubleshooting Layer 1, Layer 2, and Layer 3
Connectivity Issues
ƒ Troubleshooting Spanning Tree Protocol
ƒ Security
ƒ Common Issues for High CPU Utilization

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved. 5

Presentation_ID.scr
 Troubleshooting
Layer 1, Layer 2
and Layer 3
Connectivity Issues

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11

Define Problems

Si Si

Core Core
Switch Switch

Si

Workstation A Access Access Workstation B

Switch Distribution Switch
Switch

ƒ Performance: latency, jitter, packet loss

ƒ Connectivity: link, reachability
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12

© 2006, Cisco Systems, Inc. All rights reserved. 6

Presentation_ID.scr
 Troubleshooting Methodology
HSRP Active HSRP Active
Even Vlans Odd Vlans

Si Si

Core Core
Switch Switch

Si

Workstation A Access Access Workstation B

Switch Distribution Switch
Switch

ƒ Define issue between two specific stations

ƒ Determine path of respective packets
ƒ Begin systematic examination of path devices
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

Troubleshooting Layer 1

ƒ Connectivity
Do we have a link?

ƒ Traffic Si
Are packets passing?
Distribution
How many? Switch

ƒ Speed/duplex
Do both sides match? Access
Switch

Workstation A

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved. 7

Presentation_ID.scr
 Link Comes up for 10/100 Mbs
but Not for 1000Mbps

ƒ Is one of the four pairs in a category 5 cable broken?

ƒ The Time Domain Reflectometry (TDR) test can be run
without having to disconnect the cables to determine if
there are any broken wires in them
ƒ Helps network administrator to discriminate between
cables that can support the upgrade to higher speed
and the ones that cannot
ƒ TDR support is available for copper ports at this time,
no support for optical as of today

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

Cable Fault—Time Domain

Reflectometry (TDR)

ƒ TDR determines cable faults

ƒ Cat 5 cable has four cable pairs
ƒ TDR detects faults in cable pairs such as opens
or shorts
ƒ TDR determines position of cable fault
ƒ TDR test is invasive, link will be down for the
test duration
ƒ TDR test shows the result for each of the four
cable pairs

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved. 8

Presentation_ID.scr
 Cable Fault—TDR

Router#test cable-diagnostics tdr interface GigabitEthernet3/1

Link state may be affected during TDR test
TDR test started on interface Gi3/1
A TDR test can take a few seconds to run on an interface
Use 'show cable-diagnostics tdr' to read the TDR results.

Router#show cable-diagnostics tdr int g3/1

TDR test last run on: April 27 1:29:58

Interface Speed Pair Cable length Distance to fault Channel Pair status
--------- ----- ---- ------------------- ------------------- ------- -----------
-
Gi3/1 100 1-2 N/A N/A Pair A Terminated

3-4 N/A N/A Pair B Terminated

5-6 N/A 5 +/- 2 m Invalid Short

7-8 N/A 5 +/- 2 m Invalid Short

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

Digital Optical Monitoring (DOM)

ƒ Digital Optical Monitoring DOM is an industry-wide
standard, known as “Digital Diagnostic Monitoring
Interface for Optical Transceivers” (or SFF-8472
ftp://ftp.seagate.com/sff/SFF-8472.PDF), intended to define
a digital interface to access real-time transceivers operating
parameters such as:
Optical TX power
Optical RX power
Laser bias current
Temperature
Transceiver supply voltage

ƒ With DOM the user has capability of performing in-service

transceiver monitoring and troubleshooting operations
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18

© 2006, Cisco Systems, Inc. All rights reserved. 9

Presentation_ID.scr
 DOM Support on Cisco Transceivers
ƒ DOM capabilities is supported on selected GBIC, SFP, Xenpak,
X2 and XFP.
ƒ Refer to the DOM Compatibility Matrix for details.
ƒ The following conditions must be met for a particular transceiver
type to qualify as supported:
–Cisco engineering has successfully verified the DOM functions during
the qualification process of the transceiver.
–All the modules that Cisco has been shipping under a particular
Product ID have DOM-capable hardware.
–Cisco manufacturing tests and verifies DOM support before each
module is shipped to customers.
ƒ Sometimes not all three conditions are met and DOM commands
may work on transceivers which are not “DOM-supported.” An
example could be XENPAK-10GB-ER.

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

Digital Read-Backs Interpretation

ƒ Of the five digital diagnostic read-backs, the most relevant ones
are Optical TX and RX power as well as temperature. The
operating ranges of these three values is unique (available on the
data sheets) across all modules of the same type (e.g. all DWDM
Xenpaks).
ƒ The supply voltage is specified in the data sheet of most
transceivers. Typical values are 5V for GBICs, 3.3V for SFPs. In
10 G transceivers there are three voltage supplies 1.8, 3.3 and 5V.
Not always all three voltages are utilized, hence this information is
not called out in the data sheet.
ƒ Note that the voltage supply read-back monitors just one voltage
supply: this works on GBICs and SFPs which have one voltage
supply, but with 10G pluggables which have three separate
voltages, this parameter is not applicable.

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved. 10

Presentation_ID.scr
 Accessing DOM
ƒ transceiver type all; [no] monitoring
This command turns on/off the DOM monitoring process for all transceiver types in the system
Router(config)#transceiver type all
Router(config-xcvr-type)#monitoring

Router(config-xcvr-type)#end

ƒ DOM is accessible also via CLI interface with the “show interface transceiver” command

#show interfaces transceiver

++ : high alarm, + : high warning, - : low warning, -- : low alarm.

N/A: not applicable, Tx: transmit, Rx: receive.

mA: milliamperes, dBm: decibels (milliwatts).

Optical Optical

Temperature Voltage Current Tx Power Rx Power

Port (Celsius) (Volts) (mA) (dBm) (dBm)

----- ------- ------- ------ -------- -------- ------ ---------

Gi1/2 50.5 5.06 28.8 1.3 -9.6

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

show interface <int> transceiver detail

#show interfaces Te3/1 transceiver detail

[SKIP]
High Alarm High Warn Low Warn Low Alarm

Temperature Threshold Threshold Threshold Threshold

Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)

--------------------------------------------------------------------------------------- ---
Te3/1 31.6 79.1 74.1 4.1 -0.8

With 10 GE Interfaces the Voltage Threshold Threshold Threshold Threshold

Value Is Usually 0, Because Port (Volts) (Volts) (Volts) (Volts) (Volts)
There the Voltage Supply Is ------------------------------------------------------------------------------------------
Not Unique Unlike in GBICs Te3/1 0.00 0.00 0.00 0.00 0.00
and SFPs
Current Threshold Threshold Threshold Threshold

Port (milliamperes) (mA) (mA) (mA) (mA)

---------------------------------------------------------------------------------------------
Te3/1 99.2 130.0 130.0 20.0 10.0

Optical
Transmit Power Threshold Threshold Threshold Threshold

Port (dBm) (dBm) (dBm) (dBm) (dBm)

-------------------------------------------------------------------------------------------
Te3/1 -3.3 3.5 3.0 -1.0 -1.5

Optical
Receive Power Threshold Threshold Threshold Threshold

Port (dBm) (dBm) (dBm) (dBm) (dBm)

--------------------------------------------------------------------------------------------
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public Te3/1 -28.5 -- -6.5 -7.0 -24.1 -24.5 22

© 2006, Cisco Systems, Inc. All rights reserved. 11

Presentation_ID.scr
 Is Physical Interface Up?
Troubleshooting Layer 1
IOS# show interface GigabitEthernet 1/1
GigabitEthernet1/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is 0009.435f.8300 (bia 0009.435f.8)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is SX
output flow-control is off, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queuing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 89000 bits/sec, 141 packets/sec Are Input/Output
5 minute output rate 23000 bits/sec, 24 packets/sec Counters
226241448 packets input, 14733424090 bytes, 0 no buffer
Incrementing
Received 224084097 broadcasts (201828280 multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected Check for Any
35622 packets output, 5452233 bytes, 0 underruns errors/crc/collisions
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

Symptoms of Port Start-Up Delay

ƒ Dynamic Host Configuration

Protocol (DHCP) address is
not resolved
ƒ 802.1x Client failing or
delayed to get authenticated

Category 5 Cable

Is It a Physical
Layer Issue?
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved. 12

Presentation_ID.scr
 Port Start-up Delay—
Problem and Solution

ƒ On linkup it takes up to 30–45 seconds for

packets to flow
ƒ Three things contribute to delay in packet
forwarding on link up
Spanning Tree
Trunk auto-negotiation
Channel auto-negotiation

IOS(config)#interface range fastethernet 2/1 - 48

IOS(config-if)#switchport host
switchport mode will be set to access
spanning-tree portfast will be enabled
channel group will be disabled
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

Check Duplex Setting and Verify

Topology—Layer 1 Troubleshooting

Router#show cdp neighbors detail

-------------------------
Device ID: 6500
Entry address(es):
IP address: 10.205.0.1
Platform: cisco WS-C6506, Capabilities: Router Switch IGMP
Interface: GigabtEthernet3/1, Port ID (outgoing port): GigabitEthernet2/1
Holdtime : 138 sec

Version :
Cisco Internetwork Operating System Software
IOS (tm) s3223_rp Software (s3223_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(18)SXF7
2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Thu 19-Jan-06 02:44 by dchih

advertisement version: 2
VTP Management Domain: 'Cisco'
Native VLAN: 1
Duplex: full

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved. 13

Presentation_ID.scr
 Troubleshooting Layer 2

ƒ Trunk
Desirable | ON?

ƒ Channel Si
Desirable | ON?
Distribution
ƒ Bridge table Switch

MAC address
learned correctly? Access
Switch
ƒ Spanning Tree
Ports forwarding as expected?
Workstation A

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

Trunk—Problem
Trunk Fails to Form

ƒ A trunk is a link between

two devices that carries
multiple VLANs Si Si
simultaneously

?
‘Tr

ISL—Inter-Switch Link;
un

Cisco proprietary
k’ ‘

IEEE 802.1q—standards- X
No

based trunk encapsulation

Tru
nk

ƒ Endpoint mismatch
’

ƒ Inconsistent DTP
configuration

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved. 14

Presentation_ID.scr
 Dynamic Trunk Protocol (DTP)
ƒ What is DTP?
Automates ISL/802.1Q trunk configuration; operates between switches
Does not operate on routers; not supported on 2900XL or 3500XL
ƒ DTP synchronizes the trunking mode on link ends (i.e., native VLAN
mismatch, VLAN range mismatch, encapsulation, etc.)
ƒ DTP state on ISL/dot1Q trunking port can be set to “auto”, “on”, “off”,
“desirable”, or “non-negotiate”
ƒ Runs over link layer; assumes point-to-point link
ƒ DTP destination mac address is 01-00-0C-CC-CC-CC
ƒ Port should be able to operate as an access port—
to fall back to access mode
ƒ During negotiation do not participate in STP
ƒ VLAN1 should be added to trunk; in ISL DTP pkts send on VLAN1 and for
access or 802.1Q on native vlan
ƒ The HDLC protocol type for DTP is 0x2004 which is the SNAP format

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

DTP Packet Capture

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved. 15

Presentation_ID.scr
 Trunk—Solution
Trunking Modes

Forms Forms Forms Forms Forms Trunk

Uses DTP Trunk Trunk Trunk with Trunk with No
with Off with Auto Desirable with On Negotiate

Off No No No No No No

Auto Yes No No Yes Yes No

Desirable Yes No Yes Yes Yes No

On Yes No Yes Yes Yes Yes

No Negotiate No No No No Yes Yes

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

Trunk—Problem
Trunk Fails to Form
ƒ Take help of CDP to verify topology
ƒ One side configured for non-negotiate and other side desirable
Router#show cdp neighbors detail
-------------------------
Device ID: 6500
Entry address(es):
IP address: 10.205.0.1
Platform: cisco WS-C6506, Capabilities: Router Switch IGMP
Interface: GigabitEthernet3/1, Port ID (outgoing port): FastEthernet2/1
Holdtime : 138 sec
Version :
Cisco Internetwork Operating System Software
IOS (tm) s3223_rp Software (s3223_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(18)SXF
2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Thu 19-Jan-06 02:44 by dchih

advertisement version: 2
VTP Management Domain: 'Cisco'
Native VLAN: 1
Duplex: full

Router#show interfaces gigabitEthernet 3/1 trunk

Port Mode Encapsulation Status Native vlan

Gi3/1 desirable 802.1q not-trunking 1

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32

© 2006, Cisco Systems, Inc. All rights reserved. 16

Presentation_ID.scr
 Trunk—Commands
Show Interfaces Switchport (Cisco IOS)

ƒ show interfaces <int> switchport

ƒ show interfaces trunk
Router#sh int g3/1 trunk

Port Mode Encapsulation Status Native vlan

Gi3/1 desirable 802.1q trunking 1

Port Vlans allowed on trunk

Gi3/1 1-4094

Port Vlans allowed and active in management domain

Gi3/1 1-58,60-899,902-998,1000-1001

Port Vlans in spanning tree forwarding state and not pruned

Gi3/1 1-58,60-899,902-998,1000-1001

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

Channel—Problems
Channel Fails to Form

ƒ A channel is a method
of grouping multiple
physical links between
two devices into a
single logical link Si Si

EtherChannel® (PAgP)— EtherChannel

Cisco proprietary port
channeling
IEEE 802.3ad (LACP)—
standards-based port
channeling
ƒ Incorrect configuration
ƒ Port is err-disabled
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved. 17

Presentation_ID.scr
 Mix of Modes Allowing
PAGP to Form Channel

Switch A Switch B Result

AUTO AUTO No EtherChannel group created

AUTO DESIRABLE EtherChannel group created

DESIRABLE AUTO EtherChannel group created

DESIRABLE DESIRABLE EtherChannel group created

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

Mix of Modes Allowing

LACP to Form Channel

Switch A Switch B Result

PASSIVE PASSIVE No EtherChannel group created

PASSIVE ACTIVE EtherChannel group created

ACTIVE PASSIVE EtherChannel group created

ACTIVE ACTIVE EtherChannel group created

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved. 18

Presentation_ID.scr
 Channel—Solution
Channel Modes

Forms
Uses Forms Forms Forms
Channel
z PAgP or Channel Channel Channel
with
LACP with Off with Auto with On
Desirable

Off No No No No No

Auto
Yes No No Yes No
(Passive)
Desirable
Yes No Yes Yes No
(Active)

On No No No No Yes

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

Channel—Problems
Channel Fails to Form

ƒ Misconfiguration—auto on one side and on on the other

2w3d: %PM-4-ERR_DISABLE: channel-misconfig error detected on Po10, putting Gi4/9 in err-disable state
2w3d: %EC-5-UNBUNDLE: Interface GigabitEthernet4/9 left the port-channel

C4510-B#show etherchannel summary

Flags: D - down P - in port-channel
I - stand-alone s - suspended
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

Number of channel-groups in use: 1

Number of aggregators: 1

Group Port-channel Protocol Ports

------+-------------+-----------+-----------------------------
10 Po10(SD) - Gi4/9(D) Gi4/10(D)
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved. 19

Presentation_ID.scr
 Channel—Commands
Show Interfaces EtherChannel (Cisco IOS)
ƒ show interfaces port-channel <1-269> etherchannel

IOS#show interfaces port-channel 1 etherchannel

Age of the Port-channel = 00d:00h:03m:10s
Logical slot/port = 14/1 Number of ports = 2
GC = 0x00010001 HotStandBy port = null
Passive port list = Fa3/45 Fa3/46
Port state = Port-channel L3-Ag Ag-Inuse

Ports in the Port-channel:

Index Load Port EC state

----------+------------+------------+--------------
0 55 Fa3/45 desirable-sl
1 AA Fa3/46 desirable-sl

Time since last port bundled: 00d:00h:02m:49s Fa3/46

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

Troubleshooting Layer 2: EtherChannel

IOS# show etherchannel load-balance
Source XOR Destination IP address
Native#

IOS-cat6k# remote login switch

Trying Switch ...
Entering CONSOLE for Switch
Type "^C^C^C" to end this session

test etherchannel load-balance interface port-channel number {ip | l4port |

mac} [source_ip_add | source_mac_add | source_l4_port] [dest_ip_add |
dest_mac_add | dest_l4_port]

IOS-cat6k-sp# test etherchannel load-balance interface port-channel 1 ip

1.1.1.1 2.2.2.2
Would select Gi1/1 of Po1

IOS-cat4k# show platform software etherchannel port-channel 1 map ip 1.1.1.1

2.2.2.2
Map port for Ip 1.1.1.1, 2.2.2.2 is Gi1/1(Po1)
NOTE: Software forwarded traffic will use Gi1/1(Po1)

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved. 20

Presentation_ID.scr
 Making Sure Spanning Tree Is
Forwarding Vlan on Right Interface
IOS# show spanning-tree interface gigabitEthernet 1/1

Vlan Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------
VLAN0001 Root FWD 3 128.833 P2p

Router#sh int g3/1 trunk

Port Mode Encapsulation Status Native vlan

Gi3/1 desirable 802.1q trunking 1

Port Vlans allowed on trunk

Gi3/1 1-4094

Port Vlans allowed and active in management domain

Gi3/1 1-58,60-899,902-998,1000-1001

Port Vlans in spanning tree forwarding state and not pruned

Gi3/1 1-58,60-899,902-998,1000-1001

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Am I Seeing Mac Address on Correct

Interface? Layer 2: Bridging
IOS# show mac-address-table dynamic interface port-channel 1
Codes: * - primary entry

vlan mac address type learn qos ports

------+----------------+--------+-----+---+--------------------------
* 1 0001.c912.7bff dynamic No -- Po1

IOS# show mac-address-table ?

address address keyword
aging-time aging-time keyword
count count keyword
dynamic dynamic entry type
interface interface keyword
module display entries in DFCcard
multicast multicast info for selected wildcard
static static entry type
vlan vlan keyword
| Output modifiers
<cr>

IOS# show spanning-tree interface gigabitEthernet 1/1

Vlan Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------
VLAN0001 Root FWD 3 128.833 P2p

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved. 21

Presentation_ID.scr
 Troubleshooting Layer 3: Route/ARP

IOS# show ip route 162.123.74.1

Routing entry for 162.123.74.0/24
Known via "eigrp 1", distance 170, metric 130816, type external
Redistributing via eigrp 1
Last update from 10.1.1.1 on Vlan1, 00:01:13 ago
Routing Descriptor Blocks:
* 10.1.1.1, from 10.1.1.1, 00:01:13 ago, via Vlan1
Route metric is 130816, traffic share count is 1
Total delay is 5010 microseconds, minimum bandwidth is 1000000 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 1

IOS# show ip arp 10.1.1.1

Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.1.1.1 4 0001.c912.7bfc ARPA Vlan1

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

Troubleshooting: Useful Tools

IOS# ping
Protocol [ip]:
Target IP address: 10.1.1.1
Repeat count [5]: 1
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 10.1.1.2
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]: r
Number of hops [ 9 ]: 3
Loose, Strict, Record, Timestamp, Verbose[RV]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
Packet has IP options: Total option bytes= 15, padded length=16
Record route: <*>
(0.0.0.0)
(0.0.0.0)
Reply to request 0 (1 ms). Received packet has options
Total option bytes= 16, padded length=16
Record route:
(10.1.1.2)
(10.1.1.1)
(10.1.1.1) <*>
End of list
Success rate is 100 percent (1/1), round-trip min/avg/max = 1/1/1 ms
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved. 22

Presentation_ID.scr
 Path of Packet
Troubleshooting: Useful Tools
IOS# ping 14.18.3.200

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 14.18.3.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
IOS-cat4k# traceroute mac ip 14.18.3.20 14.18.3.200
Translating IP to mac .....
14.18.3.20 => 0009.435f.86ff
14.18.3.200 => 0003.6b73.9aff

Source 0009.435f.86ff found on IOS-cat4k

IOS-cat4k (14.18.3.20 ) : Vl1 => Gi1/1
Destination 0003.6b73.9aff found on IOS-cat4k
Layer2 trace completed.
IOS-cat4k#
IOS-cat4k# traceroute ?
WORD Trace route to destination address or hostname
appletalk AppleTalk Trace
clns ISO CLNS Trace
ip IP Trace
ipx IPX Trace
mac Trace Layer2 path between 2 endpoints
oldvines Vines Trace (Cisco)
vines Vines Trace (Banyan)
<cr>

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

Path of Packet
Troubleshooting Summary

ƒ Baseline applications
Define endpoints
Map expected path
Si
Know features in path

ƒ Change control Distribution

Switch

ƒ Apply methodical process

Access
Switch

Workstation A

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46

© 2006, Cisco Systems, Inc. All rights reserved. 23

Presentation_ID.scr
 What Caused VLANs to
Disappear from My Network?
What Is Virtual Trunking Protocol (VTP)?
ƒ Purpose: create/delete VLANs on a centralized
switch (server) and have leaf (client) switches
learn information
ƒ Runs only on trunks
ƒ Four modes:
Server: updates clients/servers—stores VLAN info in NVRAM
Client: receive updates—cannot make changes
Transparent: lets updates pass through
Off: VTP turned off

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

What Is VTP Configuration Rev. No?

VTP Configuration Revision Number Increments
for Each VLAN Change

VTP Domain A
Rev X Rev X+1
VLAN A VLAN A
VLAN B VLAN B
VLAN C Si VLAN X
VTP

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved. 24

Presentation_ID.scr
 Aha! Now I Know What Happened
VTP Bomb Occurs when a VTP Server with a Higher Revision of the
VTP Database (Albeit Loaded with Potentially Incorrect Information) Is
Inserted into the Production VTP Domain Causing the Loss of VLAN
Information on All Switches in that VTP Domain

VTP Domain A
Rev X Rev X+1
VLAN A VLAN A
VLAN B VLAN B
VLAN C Si VLAN X
VTP

Rev X+1 VTP Domain A Rev X+1

VLAN A VLAN A
VLAN B VLAN B
VLAN X Si Si VLAN X

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

VTP—Commands
Show VTP Status (Cisco IOS)

Native#show vtp status

VTP Version : 2
Configuration Revision : 5
Maximum VLANs supported locally : 1005
Number of existing VLANs : 6
VTP Operating Mode : Server
VTP Domain Name : mydomain
VTP Pruning Mode : Disabled
VTP V2 Mode : Disabled
VTP Traps Generation : Disabled
MD5 digest : 0xE3 0xE9 0x3A 0x43 0x69 0x2A 0x59
Configuration last modified by 127.0.0.12 at 2-23-02 21:43:44
Local updater ID is 10.118.2.159 on interface Vl1
(lowest numbered VLAN interface found)

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved. 25

Presentation_ID.scr
 VTP—Problem
How Can We Avoid This?
ƒ Reset the configuration revision using domain name
ƒ Change the VTP domain of the new switch to a bogus and non-
existent VTP domain name, and then change the VTP domain
back to the original name
WS-4507#show vtp status
VTP Version :2
Configuration Revision :7
Maximum VLANs supported locally : 4094
Number of existing VLANs : 16
VTP Operating Mode : Server
VTP Domain Name : Networkers2007

WS- 4507(config)#vtp domain test

Domain name set to test.

WS-4507#show vtp status

VTP Version :2
Zero-ize when
Configuration Revision :0 Change
Maximum VLANs supported locally : 255 Domain Name
Number of existing VLANs : 16
VTP Operating Mode : Server
VTP Domain Name : test

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

VTP—Problem
How Can Avoid This?
ƒ Reset the configuration revision using VTP mode
ƒ Change the VTP type from server (the default) to transparent,
and then change the mode back to client or server
WS-6500#show vtp status
VTP Version :2
Configuration Revision :4
Maximum VLANs supported locally : 4094
Number of existing VLANs : 20
VTP Operating Mode : Server
VTP Domain Name : Networkers
.
WS-6500(config)#vtp mode transparent
Setting device to VTP TRANSPARENT mode.

WS-6500(config)#vtp mode server Zero-ize When change

Setting device to VTP SERVER mode mode from server to
WS-6500#show vtp status transparent and back
VTP Version :2 to server
Configuration Revision :0
Maximum VLANs supported locally : 255
Number of existing VLANs : 20
VTP Operating Mode : Server
VTP Domain Name : Networkers
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved. 26

Presentation_ID.scr
 Agenda

ƒ Session Overview
ƒ Troubleshooting Layer 1, Layer 2, and Layer 3
Connectivity Issues
ƒ Troubleshooting Spanning Tree Protocol
ƒ Security
ƒ Common Issues for High CPU Utilization

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Troubleshooting
Spanning Tree Protocol

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved. 27

Presentation_ID.scr
 Spanning Tree Protocol
Troubleshooting Methodology

ƒ Start now—be proactive

Layer2 Loops
ƒ Divide and conquer
ƒ Document Spanning Si Si Si Si Si Si

Tree topology
ƒ Implement Spanning Layer 3
Equal Cost
Layer 3
Equal Cost
Tree enhancement Links Si Si Links
features
ƒ Develop recovery plan
Si Si Si Si

Si Si

to include data collection

for root cause analysis
WAN Data Center Internet

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

Spanning Tree Protocol

Documenting Spanning Tree Topology

Corporate Network
Gig 0/1 Gig 0/1
155.3.77.254 155.3.76.254
VLAN187 VLAN186
1/1 Root: 1,3,5,7,9,187 1/1 Root: 2,4,6,8,186
0001.c912.7800 0003.6b73.9700

DIST-01 F: 1,2,3,4,5,6,7,8,9 DIST-02

192.168.1.1 3/1–4 3/1–4 192.168.1.2

F: 1,2,3,4,5,6,7,8,9
4/4
F: 1,8,9 4/1
4/3 F: 1,2,3
4/1 4/2 F: 1,6,7 4/2 4/3 4/4
F: 1,2,3 F: 1,4,5 F: 1,4,5 F: 1,6,7 F: 1,8,9

B: 2 B: 4 B: 1,5 B: 6 B: 1,7 B: 8 B: 1,9

1/1 1/2 1/1 1/2 1/1 1/2 1/1 1/2
B: 1,3

0001.c967.7800 0001.c9dd.7800 0001.c932.9700 0001.c949.9700

Closet1 Closet2 Closet3 Closet4
192.168.1.11 192.168.1.12 192.168.1.13 192.168.1.14

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved. 28

Presentation_ID.scr
 Spanning Tree Best Practice
“How Can I Have a Spanning Tree Loop?
I Don’t Have Spanning Tree Enabled?”
ƒ Cisco recommends leaving STP-enabled for the
following reasons:
If there is a loop (induced by mispatching, bad cable, and so on),
STP will prevent detrimental effects to the network caused by multicast
and broadcast data
Protection against an EtherChannel breaking down
Most networks are configured with STP, giving it maximum field
exposure; more exposure generally equates to stable code
Protection against dual attached NICs misbehaving
(or bridging enabled on servers)
Bridging between wired and wireless
The software for many protocols (such as PAgP, IGMP snooping, and
trunking) is closely related to STP; running without STP may lead to
undesirable results

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

Spanning Tree Standards and Features

Spanning Tree Toolkit, 802.1D, 802.1s, 802.1w

ƒ 802.1D/1998: legacy standard for bridging and Spanning Tree (STP)

ƒ 802.1D/2004: updated bridging and STP standard; includes 802.1s, 802.1t,
and 802.1w
ƒ 802.1s: Multiple Spanning Tree Protocol (MSTP)—maps multiple VLANs into the
same Spanning Tree instance
ƒ 802.1t: MAC address reduction/extended system ID—moves some BPDU
bits to high-numbered VLANs from the priority field, which constrains the possible
values for bridge priority; unique “MAC” per chassis not port
ƒ 802.1w: Rapid Spanning Tree Protocol (RSTP)—improved convergence
over 1998 STP by adding roles to ports and enhancing BPDU exchanges
ƒ Cisco Features: Per VLAN Spanning Tree (PVST), PVST+, Rapid PVST, Rapid-
PVST+, UplinkFast, BackboneFast, BPDU Guard, RootGuard, LoopGuard, UDLD
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved. 29

Presentation_ID.scr
 Spanning Tree Features
ƒ PortFast*: bypass listening-learning Root Distribution
phase for access port Switches
ƒ UplinkFast: three to five seconds F F
convergence after link failure Si Si

ƒ BackboneFast: cuts convergence time by

Max_Age for indirect failure F F
ƒ LoopGuard*: prevents alternate
or root port from becoming designated in
absence of BPDUs
ƒ RootGuard*: prevents external switches X
from becoming root Wiring
F B Closet
ƒ BPDUGuard*: disable PortFast enabled
Switch
port if a BPDU is received
ƒ BPDUFilter*: do not send or receive
BPDUs on PortFast-enabled ports

*Also Supported with MST and Rapid PVST+

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

What Is Root Guard?

ƒ Root guard forces a
Layer 2 LAN interface to
be a designated port, and
if any device accessible
through the interface
becomes the root bridge,
root guard puts the Root
interface into the root-
inconsistent (blocked) state

Router(config-if)# switchport
Router(config-if)# spanning-tree guard root

%SPANTREE-2-ROOTGUARDBLOCK: Port 3/3 tried to become non-designated in VLAN 800.

Moved to root-inconsistent state

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved. 30

Presentation_ID.scr
 What Is BPDU Guard?
ƒ PortFast BPDU guard can prevent
loops by moving PortFast-configured
interfaces that receive BPDUs to
errdisable, rather than running
Spanning Tree across that port
ƒ This keeps ports configured with X
PortFast from being incorrectly
connected to another switch Or

Router(config-if)#spanning-tree portfast
Router(config-if)#spanning-tree bpduguard enable

1w2d: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port FastEthernet3/1 with BPDU Guard

enabled. Disabling port.
1w2d: %PM-4-ERR_DISABLE: bpduguard error detected on Fa3/1, putting Fa3/1 in err-disable state

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

UplinkFast
Root
ƒ Spanning Tree enhancement to
reduce failover convergence time
ƒ Used when recovery path is F F
Si Si
known and predictable
ƒ Enabled on access switch F F

ƒ Bypasses ‘listening’ and 2

‘learning’ stages of STP
ƒ Reduces failover time to 2–3
seconds from 30 seconds F B
ƒ Auto-populates upstream
address tables (dummy mcast)
ƒ Default in RSTP 1

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved. 31

Presentation_ID.scr
 BackboneFast
Root
ƒ Spanning Tree
enhancement to reduce 3
F F
failover convergence time Si Si

ƒ Targeted at indirect failures F F

ƒ Enabled on all switches
ƒ Bypasses ‘max-age’
ƒ Reduces failover time F B
to 30 seconds from
50 seconds
ƒ Default in RSTP

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

802.1s(MST) Overview
ƒ Two active topologies VLANs 1–50:
Instance 1
VLANs 51–99:
Instance 2
ƒ All VLANs mapped to one
of two topologies Si Si

ƒ Lower BPDU counts

ƒ Much less CPU utilization
ƒ Very high scalability
ƒ 802.1s: 12.1(11)EX
ƒ Reduces complexity of F B B F
numerous topologies

The Problem with Running a Single Instance of STP Is

That Any Blocked Link Is Unable to Actively Participate in the
Forwarding of Data—thus It Becomes a Wasted Resource—
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved. 32

Presentation_ID.scr
 Spanning Tree Issues

ƒ 802.1D-based Spanning Tree implementations don’t

converge fast (2 x Fwd_Delay + Max_Age)
ƒ Traditional Spanning Tree is based on
network-wide timers
ƒ Cisco’s PortFast, UplinkFast, and BackboneFast help,
but standardization would be better
ƒ IEEE work resulted in new standard: Rapid Spanning
Tree Protocol (RSTP), defined in 802.1w

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

RSTP(802.1w) Overview
ƒ Takes advantage of today’s
topologies (full-duplex Root
point-to-point links)
ƒ No more network-wide timers
when all switches run 802.1w Proposal Agreement
ƒ Handshake mechanism 1 2
between bridges
ƒ Proposal-agreement messaging Proposal Agreement
(“I want to become designated—
do you agree?”) 3 4

ƒ Can achieve subsecond

convergence

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved. 33

Presentation_ID.scr
 RSTP Overview (Cont.)

ƒ Incorporates mechanisms
similar to UplinkFast/ Root
BackboneFast extensions
ƒ Decouples port status/role Proposal Agreement
(i.e., forwarding designated) 1 2

ƒ No need to tune timers

ƒ Backwards compatible Regular 802.1D
BPDUs
with 802.1d/PVST+ on a
per-port basis PVST

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

Spanning Tree Protocol

Troubleshooting Commands
IOS#show spanning-tree vlan 1 brief

VLAN0001
Spanning tree enabled protocol ieee
Root ID Priority 1
Address 0060.8355.7b00
Cost 23
Port 1 (GigabitEthernet1/1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32769 (priority 32768 sys-id-ext 1)

Address 0007.0e8f.0880
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300

Interface Designated
Name Port ID Prio Cost Sts Cost Bridge ID Port ID
-------------------------- ----------- ------ ------- -------
GigabitEthernet1/1 128.1 128 4 FWD 67 32768 0005.5f33.dc01 128.1
FastEthernet3/48 128.176 128 19 FWD 48 32768 0030.7bdd.5080 128.16

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved. 34

Presentation_ID.scr
 Spanning Tree Protocol
Troubleshooting Commands

IOS#show spanning-tree vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 32768
Address 0030.7b4e.4801
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768

Address 0030.7b4e.4801
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- ------------------------
Fa2/1 Desg FWD 19 128.129 P2p Peer(STP)

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

Spanning Tree Protocol

Troubleshooting Commands

Root for
IOS#show spanning-tree summary
Root bridge for: VLAN0010. Listed
Extended system ID is enabled VLANs
PortFast BPDU Guard is enabled
EtherChannel misconfiguration guard is disabled
UplinkFast is disabled
BackboneFast is enabled
Default pathcost method used is short

Name Blocking Listening Learning Forwarding STP Active

------------------------------------- ------------- ------------- ---------------- --------
VLAN0001 0 0 0 2 2
VLAN0010 0 0 0 1 1
VLAN1002 0 0 0 1 1
VLAN1003 0 0 0 1 1
VLAN1004 0 0 0 1 1
VLAN1005 0 0 0 1 1
------------------------ ------------- ------------- ------------- ----------------

6 VLANs 0 0 0 7 7

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70

© 2006, Cisco Systems, Inc. All rights reserved. 35

Presentation_ID.scr
 Spanning Tree Protocol
Logical Ports and STP Instances Max
ƒ =(Number of non-ATM trunks Recommended
Instances
* number of VLANs on trunk)
2950 64 VLANs
ƒ +(Number of ATM trunks
* VLANs on trunk *2) 3550 128 VLANs

ƒ +Number of nontrunking ports 3750-E 128 VLANs

ƒ [(Number of active VLANs x 3560 128 VLANs

number of trunks)+ number of 4000 Sup I or II 1,500 VLANs
access ports]
4500 Sup II+,IV,V 3,000 VLANs
ƒ *VTP pruning does not remove
STP from trunks 6000 Sup I 4000 VLANs

6500 Sup II 14,000 VLANs

6500 Sup 32 11,000 VLANs

6500 Sup 720 14,000 VLANs
See Respective Platform
Release Notes for More Details
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71

Spanning Tree Protocol

Troubleshooting Commands
IOS# show proc cpu
CPU utilization for five seconds: 1%/0%; one minute: 2%; five minutes: 2%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager
<…some output removed…>
79 0 256 0 0.00% 0.00% 0.00% 0 mls-msc Process
80 30508 461976 66 0.40% 0.43% 0.44% 0 Spanning Tree
81 108 27024 3 0.00% 0.00% 0.00% 0 Ethchnl
<…some output removed…>
162 12 41 292 0.00% 0.01% 0.00% 1 Virtual Exec

IOS# show spanning-tree summary

<…some output removed…>

Name Blocking Listening Learning Forwarding STP Active

---------------------- -------- --------- -------- ---------- ----------
VLAN0001 1 0 0 1 2
<…some output removed…>
VLAN1005 0 0 0 1 1
---------------------- -------- --------- -------- ---------- ----------
282 vlans 1 0 0 282 283

Number of
Spanning Tree
Instances
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72

© 2006, Cisco Systems, Inc. All rights reserved. 36

Presentation_ID.scr
 Spanning Tree Protocol
Troubleshooting Topology Change
IOS#show spanning-tree vlan 1 detail

VLAN0001 is executing the ieee compatible Spanning Tree protocol

Bridge Identifier has priority 32768, address 0005.7495.9101
Don’t
Configured hello time 2, max age 20, forward delay 15 Forget
Current root has priority 32768, address 0001.c912.7800
Root port is 70 (GigabitEthernet2/6), cost of root path is 4 PortFast
Topology change flag not set, detected flag not set
Number of topology changes 4 last change occurred 02:17:20 ago
from Port-channel1
Times: hold 1, topology change 35, notification 2
hello 2, max age 20, forward delay 15
Timers: hello 0, topology change 0, notification 0, aging 300

Port 70 (GigabitEthernet2/6) of VLAN0001 is forwarding

Port path cost 4, Port priority 128, Port Identifier 128.70.
Designated root has priority 32768, address 0001.c912.7800
Designated bridge has priority 32768, address 0001.c912.7800
Designated port id is 128.70, designated path cost 0
Timers: message age 2, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 7, received 4162

Port 833 (Port-channel1) of VLAN0001 is blocking

Port path cost 4, Port priority 128, Port Identifier 128.833.
Designated root has priority 32768, address 0001.c912.7800
Designated bridge has priority 32768, address 0001.c912.7800
Designated port id is 128.769, designated path cost 0
Timers: message age 1, forward delay 0, hold 0
Number of transitions to forwarding state: 1
Link type is point-to-point by default
BPDU: sent 4, received 134836
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73

Spanning Tree Protocol

Troubleshooting Commands
ƒ Track down source of changes
TCN, logs, network management

ƒ Protect against the changes

UDLD, PortFast, network management

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74

© 2006, Cisco Systems, Inc. All rights reserved. 37

Presentation_ID.scr
 Can Unidirectional Link Detection (UDLD)
Help to Avoid Spanning Tree Loop?
What Is UDLD?
ƒ Detects one-way logical
connectivity
ƒ Physical-layer errors
are detected by auto- Faulty Gbic?
negotiation and FEFI*
RX TX
ƒ Detects faults at Layer 2 Si Si
TX RX

6500-1>sh int g2/1

GigabitEthernet2/1 is up, line protocol is up

*FEFI: Far-End Fault Indication

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75

Why Are Uni-Dir Links a Bad Thing?

Uni-Dir Link
ƒ Root xmits BPDUs Root
TX
RX
ƒ Neighbor doesn’t RX
receive them and thinks Si
TX
TX Si
RX
the root is dead Znow RX
TX
claims it’s the new root
UDLD
ƒ Bottom switch opens up
its blocked port Xloop in RX TX
the network TX RX
ƒ Network goes down, Si
troubleshooting very
difficult RX

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76

© 2006, Cisco Systems, Inc. All rights reserved. 38

Presentation_ID.scr
 Show UDLD
IOS# show udld gigabitEthernet 1/1
Interface Gi1/1
---
Port enable administrative configuration setting: Enabled / in aggressive mode
Port enable operational state: Enabled / in aggressive mode
Current bidirectional state: Bidirectional
Current operational state: Advertisement - Single neighbor detected
Message interval: 15
Time out interval: 5

Entry 1
---
Expiration time: 35
Device ID: 1
Current neighbor state: Bidirectional
Device name: SAL06090FCJ
Port ID: Gi1/1
Neighbor echo 1 device: SAD044204Y8
Neighbor echo 1 port: Gi1/1

Message interval: 5
CDP Device name: ls-7603-16a

%PM-4-ERR_DISABLE: udld error detected on Gi1/0/25, putting Gi1/0/25 in err-disable state

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

Spanning Tree: Commands

UDLD Enable/Aggressive
ƒ Native can have standard or aggressive configured
globally and per port exceptions

IOS(config)#udld enable
IOS(config)#interface gigabitEthernet 1/1
IOS(config-if)#udld aggressive

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78

© 2006, Cisco Systems, Inc. All rights reserved. 39

Presentation_ID.scr
 Spanning Tree Protocol
STP Loop Recovery
ƒ Do not power off switches—pull/shut redundant links
ƒ If possible, initially disable ports that should be blocking
ƒ Check and physically remove the connections to the
ports that should be blocking
ƒ Set up remote access to your network and call TAC

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79

Spanning Tree Protocol

Troubleshooting Summary
ƒ Be proactive!
Layer 2 Loops
ƒ Use the diagram of
the network
Si Si Si Si Si Si

ƒ Know where the root is

ƒ Know where redundancy is
Layer 3 Layer 3
ƒ Minimize the number of Equal Cost Equal Cost
blocked ports Links Si Si Links

ƒ Keep STP even if it is

unnecessary Si Si Si Si

Si Si

ƒ Have modem access

to key devices, call TAC

WAN Data Center Internet

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80

© 2006, Cisco Systems, Inc. All rights reserved. 40

Presentation_ID.scr
 Agenda

ƒ Session Overview
ƒ Troubleshooting Layer 1, Layer 2, and Layer 3
Connectivity Issues
ƒ Troubleshooting Spanning Tree Protocol
ƒ Troubleshooting Security
ƒ Troubleshooting High CPU Utilization

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81

Troubleshooting
Security

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82

© 2006, Cisco Systems, Inc. All rights reserved. 41

Presentation_ID.scr
 Port Security
ƒ What it does:
Limits the number of MAC addresses that are able to connect to
a switch and ensures only approved MAC addresses are able to
access the switch
ƒ Benefit:
Ensures only approved users can log on to the network

Valid MAC Address

√
X
Invalid MAC Address
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83

Port Security Details

ƒ Configuration options
Interface FastEthernet1/1
switchport port-security
switchport port-security maximum 3
switchport port-security aging time 1
switchport port-security violation restrict
switchport port-security aging type inactivity

• Default action—shutdown
1w2d: %PM-4-ERR_DISABLE: psecure-violation error detected on Fa3/1, putting Fa3/1 in err-disable state
1w2d: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred,
caused by MAC address 0005.dccb.c941 on port FastEthernet3/1.

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84

© 2006, Cisco Systems, Inc. All rights reserved. 42

Presentation_ID.scr
 Port Security Details
Switch_B#sh port-security
Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action
(Count) (Count) (Count)
---------------------------------------------------------------------------
Fa3/5 3072 3072 0 Restrict
Fa3/10 10 2 19172 Restrict
---------------------------------------------------------------------------
Total Addresses in System (excluding one mac per port) : 3072
Max Addresses limit in System (excluding one mac per port) : 3072
Switch_B#

Switch_B#sh port-security
Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action
(Count) (Count) (Count)
---------------------------------------------------------------------------
Fa3/5 3072 3072 0 Restrict
Fa3/10 10 2 19172 Restrict
---------------------------------------------------------------------------
Total Addresses in System (excluding one mac per port) : 3072
Max Addresses limit in System (excluding one mac per port) : 3072
Switch_B#

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 85

Understanding 802.1x
How It Works
ƒ Each person trying to enter the network must
receive authorization based on their personal
username and password
Valid Username
Valid Password

Yes
TACACS+ or
RADIUS No

Invalid Username
Invalid Password

Client Accessing TACACS+ or RADIUS

Switch Server
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 86

© 2006, Cisco Systems, Inc. All rights reserved. 43

Presentation_ID.scr
 Understanding 802.1x
! enable AAA Switch#sh dot1x
aaa new-model Sysauthcontrol = Enabled
! use AAA for 802.1x only (optional) Dot1x Protocol Version = 1
aaa authentication login default none Dot1x Oper Controlled Directions =
aaa authentication dot1x default group radius Both
! set IP address of radius server Dot1x Admin Controlled Directions =
radius-server host 10.48.66.102 Both
! radius server key
radius-server key Cisco Switch#sh dot1x interface g2/16
! enable 802.1x AuthSM State = HELD
dot1x system-auth-control BendSM State = IDLE
! L3 interface for accessing RADIUS server PortStatus = UNAUTHORIZED
interface Vlan1 MaxReq = 2
ip address 10.48.72.177 255.255.254.0 MultiHosts = Disabled
! RADUIS server is behind this L2 port Port Control = Auto
interface gi2/1 QuietPeriod = 60 Seconds
switchport Re-authentication = Disabled
switchport mode access ReAuthPeriod = 3600 Seconds
switchport access vlan 1 ServerTimeout = 30 Seconds
! enable 802.1x on the interface SuppTimeout = 30 Seconds
interface gi2/16 TxPeriod = 30 Seconds
switchport
switchport mode access
dot1x port-control auto ƒ Debugging commands:
end debug dot1x event
debug radius
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 87

PC Is Authenticated in Correct Vlan but

Have IP Address from DHCP in Guest Vlan
ƒ Tx-period: Default is 30 sec; switch expects response from client
before retransmitting EAP-Identity-Request frame again
ƒ Max-reauth-req: Default is 2
ƒ Configuring the minimum values, a switch port can be deployed
into the guest VLAN in 5 seconds if our timers are very aggressive
ƒ DHCP and the 802.1x processes are completely asynchronous

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 88

© 2006, Cisco Systems, Inc. All rights reserved. 44

Presentation_ID.scr
 DHCP Snooping
ƒ What it does:
DHCP
Switch forwards only DHCP DHCP Snooping-Enabled Server
requests from untrusted
access ports, drops all other Trusted
Si

st
types of DHCP traffic; allows

ue
only designated DHCP ports

eq

DH
or uplink ports trusted to relay

CP
P
DHCP messages
X

HC

AC
Untrusted

D
Builds a DHCP binding table

K
containing client IP address,
client MAC address, port,
VLAN number

ƒ Benefit:
DHCP
Eliminates rogue devices from Client
behaving as the DHCP server Rogue
Server

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 89

DHCP Snooping

Switch(config)# ip dhcp snooping

Switch(config)# ip dhcp snooping vlan 10 100
Switch(config)# int f6/1
Switch(config-if)# ip dhcp snooping trust
Switch(config-if)# ip dhcp snooping limit rate <rate>

Switch# show ip dhcp snooping

Switch DHCP snooping is enabled
DHCP snooping is configured on following VLANs:1
Insertion of option 82 is enabled
Interface Trusted Rate limit (pps)
-------------------- ------- ----------------
FastEthernet2/1 yes 100

Switch# show ip dhcp snooping binding

MacAddress IpAddress Lease(sec) Type VLAN Interface
------------------ --------------- ---------- ------- ---- ----------------
0000.0100.0201 10.0.0.1 1600 dynamic 100 Fa2/1

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 90

© 2006, Cisco Systems, Inc. All rights reserved. 45

Presentation_ID.scr
 What We Should Know
Before We Start Troubleshooting?

ƒ Configured to rate-limit the incoming DHCP packets

ƒ Points to note:
DHCP request broadcasted to only trusted ports in that vlan
DHCP responses unicasted to the client port only
DHCP responses on untrusted port is dropped

ƒ Option 82 enabled by default, when dhcp snooping

is enabled
ƒ Option 82 DHCP pkt is dropped when rcvd on
untrusted port

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 91

When Upstream Switch Is a Relay Agent:

Interface Config: DHCP Server
ip dhcp relay information
option trusted

DHCP Relay Agent

(e.g. Cat6k)

Trusted Port
Cat4k as Edge Switch
(Inserts Option 82)

Untrusted Ports

Customer A

Customer B

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 92

© 2006, Cisco Systems, Inc. All rights reserved. 46

Presentation_ID.scr
 Dynamic ARP Inspection
Dynamic ARP
Inspection Protects
Against ARP Poisoning Gateway = 10.1.1.1
MAC=A Si

ƒ Uses the DHCP-snooping

binding table
ƒ Tracks MAC to IP from DHCP
transactions
ƒ Rate-limits ARP requests Gratuitous ARP
from client ports; stop 10.1.1.50=MAC_B
port scanning Gratuitous ARP
10.1.1.1=MAC_B
ƒ Drop BOGUS ARPs;
prevents ARP poisoning/
MIM attacks
Attacker = 10.1.1.25 Victim = 10.1.1.50
MAC=B MAC=C

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 93

Dynamic ARP Inspection

Switch(config)#ip arp inspection vlan 1

Switch(config)#ip arp inspection filter static-hosts vlan 1

Switch(config)#arp access-list static-hosts

Switch(config-arp-nacl)#permit ip host 10.1.1.5 mac any

Switch#show ip arp inspection vlan 1

Source Mac Validation : Disabled

Destination Mac Validation : Disabled
IP Address Validation : Disabled

Vlan Configuration Operation ACL Match Static ACL

---- ------------- --------- --------- ----------
1 Enabled Active static-hosts No

Vlan ACL Logging DHCP Logging

---- ----------- ------------
1 Deny None

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 94

© 2006, Cisco Systems, Inc. All rights reserved. 47

Presentation_ID.scr
 Dynamic ARP Troubleshooting
Switch_A# sh ip arp inspection statistics
Vlan Forwarded Dropped DHCP Drops ACL Drops
---- --------- ------- ---------- ---------
5 200 10 5 5

Vlan DHCP Permits ACL Permits Source MAC Failures

---- ------------ ----------- -------------------
5 125 75 0

Vlan Dest MAC Failures IP Validation Failures Invalid Protocol Data

---- ----------------- ---------------------- ---------------------
5 0 0 0
Switch_A#

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 95

IP Source Guard
Protection Against Spoofed IP Addresses
IP Source Guard Protects
Against Spoofed IP Gateway = 10.1.1.1
Si

Addresses
ƒ Uses the DHCP-snooping
binding table
ƒ Tracks IP address to
port associations
ƒ Dynamically programs Hey, I’m 10.1.1.50
port ACL to drop traffic not
originating from IP address
assigned via DHCP

Attacker = 10.1.1.25 Victim = 10.1.1.50

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 96

© 2006, Cisco Systems, Inc. All rights reserved. 48

Presentation_ID.scr
 IP Source Guard

Switch(config)# ip dhcp snooping vlan 10 20

Switch(config)# interface fa6/1
Switch(config-
if)# switchport trunk encapsulation dot1q
Switch(config-if)# switchport mode trunk
Switch(config-if)# switchport trunk native vlan 10
Switch(config-if)# switchport trunk allowed vlan 11-20
Switch(config-if)# no ip dhcp snooping trust
Switch(config-if)# ip verify source vlan dhcp-snooping

Switch# sh ip verify source interface f6/1

Interface Filter-type Filter-mode IP-address Mac-
address Vlan
--------- ----------- ----------- --------------- ----
------------- ----------
Fa6/1 ip-mac active 10.1.1.3
00:04:9A:49:E5:FF 10
Fa6/1 ip-mac active deny-all
11-20

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 97

Troubleshooting Commands:
IPSG in IP Mode
Switch_A# sh ip dhcp binding
IP address Client-ID/ Lease expiration Type
Hardware address
10.1.1.3 0063.6973.636f.2d30. Mar 30 2007 02:50 AM Automatic
3030.342e.3961.3439.
2e65.3566.662d.566c.
35
Switch_A# sh ip dhcp snooping binding
MacAddress IpAddress Lease(sec) Type VLAN Interface
------------------ --------------- ---------- ------------- ---- ----------
----------
00:04:9A:49:E5:FF 10.1.1.3 82522 dhcp-snooping 10 FastEthernet6/1
Total number of bindings: 1

Switch_A# sh ip verify source

Interface Filter-type Filter-mode IP-address Mac-address Vlan
--------- ----------- ----------- --------------- ----------------- -------
---
Fa6/1 ip active 10.1.1.3 10

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 98

© 2006, Cisco Systems, Inc. All rights reserved. 49

Presentation_ID.scr
 Access Control Lists

ƒ What it does:
Allows or denies access
based on the source or
destination address Types of ACLs
Restricts users to designated ƒ Router ACL (RACL)
areas of the network, blocking
unauthorized access to all other ƒ VLAN ACL (VACL)
applications and information ƒ Port-based ACL
(PACL)
ƒ Benefit:
Prevents unauthorized access
to servers and applications
Allows designated users to
access specified servers

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 99

Applying a RACL/PACL
interface Vlan4
ip address 4.4.4.1 255.255.255.0 Counters
end

Switch#show ip access-lists RACL

Extended IP access list 101
deny tcp host 200.200.200.1 any neq 80 (5 matches)
permit ip any any (11915 matches)

Switch(config)#interface vlan 4 PACL

Switch(config-if)#ip access-group 101 in
Switch(config-if)#

Switch(config)#interface fa 4/23
Switch(config-if)#swichport access vlan 4
Switch(config-if)#ip access-group 101 in

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 100

© 2006, Cisco Systems, Inc. All rights reserved. 50

Presentation_ID.scr
 VLAN ACL Map (VACL)
mac access-list extended drop-appletalk
ƒ VACLs match all packets
permit any any protocol-family
appletalk on the VLAN

ip access-list extended ip2

ƒ VACLs may have IP-
permit ip any any based and MAC-based
ACLs, with implicit deny
vlan access-map vacl-100 15 all at the end
action drop
match mac address drop-appletalk ƒ This example will permit
vlan access-map vacl-100 20 IP and drop all AppleTalk
action forward frames on VLAN 201
match ip address ip2
!
vlan filter vacl-100 vlan-list 201

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 101

Catalyst Integrated Security Features

Summary Cisco IOS
ip dhcp snooping
IP Source Guard ip dhcp snooping vlan 2-10
ip arp inspection vlan 2-10
Dynamic ARP Inspection
!
DHCP Snooping interface fa3/1
switchport port-security
Port Security switchport port-security max 3
switchport port-security violation restrict
switchport port-security aging time 2
ƒ Port security prevents MAC
flooding attacks switchport port-security aging type inactivity
ƒ DHCP snooping prevents client attack ip arp inspection limit rate 100
on the switch and server ip dhcp snooping limit rate 100
ƒ Dynamic ARP inspection adds security
no ip dhcp snooping trust
to ARP using DHCP snooping table
ƒ IP source guard adds security ip verify source vlan dhcp-snooping
to IP source address using DHCP Interface gigabit1/1
snooping table
ip dhcp snooping trust
ƒ All features work on switchports
ip arp inspection trust
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 102

© 2006, Cisco Systems, Inc. All rights reserved. 51

Presentation_ID.scr
 NAC Sessions

ƒ SEC-2041: Deploying Cisco NAC Appliance

for Diverse Access Methods
ƒ SEC-3040: Troubleshooting NAC
ƒ SEC-3041: Troubleshooting Cisco NAC Appliance
ƒ SEC-2030: Deploying Network-Based Intrusion
Prevention Systems
ƒ SEC-2031: Deploying Host-Based Intrusion Prevention
Technology
ƒ SEC-3030: Troubleshooting Intrusion
Detection Systems

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 103

Agenda

ƒ Session Overview
ƒ Troubleshooting Layer 1, Layer 2, and Layer 3
Connectivity Issues
ƒ Troubleshooting Spanning Tree Protocol
ƒ Security
ƒ Common Issues for High CPU Utilization

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 104

© 2006, Cisco Systems, Inc. All rights reserved. 52

Presentation_ID.scr
 Troubleshooting
High CPU Utilization

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 105

Common Reasons
for High CPU Utilization

ƒ Packets are process switched

ƒ If switch cannot forward packet in
hardware because fragmentation issue
ƒ Packets coming with IP options
ƒ Expired TTL
ƒ ACL configured with log keyword
ƒ ACL failed to get programmed in hardware
ƒ IP routes failed to get programmed in hardware

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 106

© 2006, Cisco Systems, Inc. All rights reserved. 53

Presentation_ID.scr
 Issues Encountered
with High CPU Utilization

ƒ Degrade performance of network

ƒ On router HSRP status may flap from active to standby
ƒ Router will lose its routing neighbors
ƒ May fail to access the switch via SSH or Telnet and
many more

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 107

Basic Commands to Understand CPU

Utilization: Know the CPU Baseline
CAT6K-STATIC#show processes cpu sorted

CPU utilization for five seconds: 71%/70%; one minute: 0%; five minutes: 0%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
239 32 59 542 0.15% 0.01% 0.00% 1 Virtual Exec
118 388712 1264243 307 0.07% 0.01% 0.00% 0 QOS Stats Export
------------- Snip------------

71% Is the average total utilization during the last 5 seconds (interrupts + processes)
70% Is the average utilization due to interrupts, during the last 5 seconds
Use show proc cpu history cmd to view a more detailed history of CPU utilization “history”

CAT6K-STATIC#show interface vlan 1

Vlan1 is up, line protocol is up
Hardware is EtherSVI, address is 000c.cf2b.9c00 (bia 000c.cf2b.9c00)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
Input queue: 0/2000/9986/890(size/max/drops/flushes); Total output drops: 0
------------- Snip------------
5 minute input rate 7890000 bits/sec, 4560 packets/sec
5 minute output rate 7500 bits/sec, 10 packets/sec

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 108

© 2006, Cisco Systems, Inc. All rights reserved. 54

Presentation_ID.scr
 Basic Commands to Understand CPU
Utilization
CAT6K-STATIC#show interface switching
Vlan1
Throttle count 0
Drops RP 0 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 63 Drops 0
Protocol Path Pkts In Chars In Pkts Out Chars Out
Other Process 6 462 0 0
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
IP Process 652 57635 603 8654
Cache misses 0
Fast 905 66904 902 53982
Auton/SSE 0 0 905 70484
ARP Process 30608 1836480 111 12432
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 109

What Should Be Our Approach

ƒ A local span session can be configured to capture the

traffic for analysis
ƒ Check log for any error messages which tell us about
resource issues
ƒ Make sure Spanning Tree is stable
ƒ We can capture traffic going to CPU with help of TAC
on Cat6500/Cat4500

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 110

© 2006, Cisco Systems, Inc. All rights reserved. 55

Presentation_ID.scr
 Storm Control Can Help to Protect CPU

ƒ Configuring traffic storm control to avoid packets flood

the LAN, creating excessive traffic and degrading
network performance
ƒ Router(config-if)# storm-control
broadcast level level[.level]
WS-C3750-24TC-L-A(config)#storm-control
broadcast level pps 1000 500

ƒ Router(config-if)# storm-control
multicast level level[.level]
WS-C3750-24TC-L-A(config-if)#storm-control
multicast level bps 100000 1000

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 111

Best Practices

ƒ Following building codes results in solid well

constructed homes and buildings
ƒ Following LAN switching “building code”
results in resilient well-constructed and stable
switched networks
ƒ Practices for Cisco Catalyst® 4500/4000, 5500/5000,
and 6500/6000 Series Switches, Running Cisco
CatOS/IOS Configuration and Management
ƒ http://www.cisco.com/warp/customer/473/103.html
ƒ http://www.cisco.com/warp/customer/473/185.html

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 112

© 2006, Cisco Systems, Inc. All rights reserved. 56

Presentation_ID.scr
 Recommended Reading

ƒ Continue your Cisco Live

learning experience with further
reading from Cisco Press
ƒ Check the Recommended
Reading flyer for suggested
books

Available Onsite at the Cisco Company Store

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 113

Complete Your Online

Session Evaluation
ƒ Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
ƒ Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
ƒ Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.

BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 114

© 2006, Cisco Systems, Inc. All rights reserved. 57

Presentation_ID.scr
 BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 115

References
http://www.cisco.com
ƒ Catalyst 4000 Troubleshooting TechNotes
http://www.cisco.com/en/US/products/hw/switches/ps663/prod_tech_notes_list.html
ƒ High CPU Utilization on Cisco IOS Software-Based Catalyst 4500 Switches
Document ID: 65591
ƒ Best Practices for Catalyst 6500/6000 Series and Catalyst 4500/4000
Series Switches Running Cisco IOS Software
Document ID: 24330
ƒ Catalyst 4500 System Message Guide
ƒ DOM Compatibility Matrix
http://www.cisco.com/en/US/docs/interfaces_modules/transceiver_modules/compatibi
lity/matrix/OL_8031.html
http://www.cisco.com/en/US/docs/interfaces_modules/transceiver_modules/compatibi
lity/matrix/OL_6974.html
http://www.cisco.com/en/US/docs/interfaces_modules/transceiver_modules/compatibi
lity/matrix/OL_6981.html
ƒ Cisco Transceiver Data Sheets
http://www.cisco.com/en/US/products/hw/modules/ps5455/products_data_sheets_list.
html
BRKRST-3131
14513_04_2008_c2 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 116

© 2006, Cisco Systems, Inc. All rights reserved. 58

Presentation_ID.scr