Troubleshooting IOS-XE

BRKSPG-3611

www.ciscolivevirtual.com
Agenda
 IOS XE and ASR1000
 Deployment Guidelines
 Troubleshooting IOS XE
Packet drops, High CPU, Memory issues Q&A
 Handle unexpected reloads
 Monitoring System Health

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
IOS XE and ASR1000
What’s IOS XE ?
 Evolution of Cisco pre-eminent IOS
Almost same extensive feature sets as IOS
Same IOS Command Line Interface

 Next Generation software platform

Kernel-based open system
Cross-platform
Modern CPU architecture
Hosted Apps and Services

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Software Architecture - IOS XE
Apps
IOSd
Hosted
Apps/Services

Processes

Linux Kernel-based
Operating system

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
 IOS XE Shipment
ASR1000 ASR903

Cat4500E/Sup7E

BRKRST-3067
Troubleshooting Catalyst 4K and 6K

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 ASR1000 Models
ASR1002 ASR1004 ASR1013

ASR1002-F ( End of Life announced )

ASR1006

ASR1001

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 ASR1000 Components
Route Processor
RP1/RP2

Enhanced Services
Processor
ESP 5/10/10-N/20/40

SPA Interface Processor

SIP10/SIP40

Shared Port Adapters

ATM/POS/Serial/Ethernet/CEOP SPA
Webex Node / DSP Service SPA
BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
RP1 and RP2

HW CPU Memory IOS XE OS Models

RP1 PowerPC 32bits ASR1002

Single Core 4G Max PPC Linux ASR1004
1.5 GHz Kernel ASR1006
RP2 Intel x86 Xeon 64bits ASR1004
Dual Core 16G Max x86 Linux Kernel ASR1006
2.66 GHz ASR1013

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Enhanced Services Processor
HW Forwarding Control Control Models
Performance CPU Memory
ESP5 8Mpps 800 MHz 1GB ASR1002

ESP10 17Mpps 800 MHz 2GB ASR1002

ASR1004
ASR1006
ESP20 24Mpps 1.2 GHz 4GB ASR1004
ASR1006
ESP40 24Mpps 1.6GHZ 8GB ASR1006
ASR1013

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 ASR 1000 Building Blocks Active Standby

Enhanced Service
Enhanced Service I2C Route
Route Processor
Processor Processor Processor
FECP RP
FECP DataLink
RP CPU
CPU
QFP
Crypto
QFP Interconn.
Crypto EOBC(1G)
Interconn.
Interconn.
Standby Interconn.
Interconn. I2C

Active SPA
IOCP EOBC(1G)
Agg.
DataLink
(Serdes)
SPA … SPA
SPA Interface Processor
BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 ASR 1000 Software Architecture Software Redundancy

FECP QFP RP IOSd IOS d

Client/Driver (Active) (Standby)

Forwarding Chassis Chassis Forwarding

Manager Manager
Interface
Manager Manager Manager

Kernel Kernel

Enhanced Service Route Processor

Processor Control Messaging

IOCP SPA
SPA
Driver SPA Interface
Chassis Interface Processor
Manager Manager
SIP
Kernel
BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
IOSd Software Redundancy

 Only for Single RP model only.

 Hardware Redundancy for dual RP models

Single IOSd with Each IOSd with

45% of total RP memory 20% of total RP memory
By software redundancy

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Deployment Guidelines
IOS Migration to IOS XE

 Feature Parity
ASR1K does not block configuration of
unsupported features
 Scale and Limitation
Multiple 72xx could be collapsed onto a smaller
number of ASR1Ks

CCO documents are good place to start !

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Unsupported Features
“ Broadband Access Aggregation and DSL Configuration Guide, Cisco
IOS XE Release 3S”

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
“ Cisco ASR 1000 Series Aggregation
Services Routers Software
Configuration Guide ”

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
“ Cisco IOS XE 3S Release Notes ”

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Software Redundancy and Limits
• “Cisco ASR 1000 Series Aggregation Services Routers Software
Configuration Guide “

• Software redundancy on RP1 will reduce Max limits and scales by half

• Software redundancy has less impact in RP2

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 ASR1K Control Plane Policing(CPP)
RP CPU  Data/Control/Mangement traffic punt to RP CPU:
 PPPoE discovery packets
Inject  Routing protocol packets
 ARPs
Control
 Multicast control traffic (e.g., IGMP, PIM)
Data

 Management traffic (e.g. FTP/Telnet/SNMP)

 Broadcast traffic
Punt
 IP fragmentation
 Packets with IP Options
ESP
QFP  Not a complete list but a baseline for thought

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 ASR1K Control Plane Policing
RP CPU
 Class-map Matches :
- IP ACL: Addr, TCP/UDP ports, Options, Fragments
Inject - IP DSCP, Prec (IPv4/v6)
- Protocol: IPv6, ARP, PPPoE data and control
Control
 Policy-map Actions:
Data

- Mark (set) and Police (rate or PPS)

Punt

(config)control-plane
service-policy input copp-policy-in
ESP service-policy output copp-policy-out
QFP

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
 ASR1K Control Plane Policing

RP CPU GE0
 Traffic from Management Interface /Console /Aux
are not subject to CPP

 Locally processed traffic in QFP:

- ICMP Echo Requests
- PPP keepalives
Punt-keepalives

 Punt-keepalives – Punt path monitor

ESP
QFP Local
Responder

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 ASR1K CPP Monitoring
 Show policy-map control-plane
RP CPU ASR-1K#sh policy-map control-plane
Control Plane
Service-policy input: CoPP
Class-map: CoPP-undesirable (match-all)
Inject
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: access-group name CoPP-undesirable-acl
Control
police:
Data

cir 8000 bps, bc 1500 bytes, be 1500 bytes

conformed 0 packets, 0 bytes; actions: transmit
exceeded 0 packets, 0 bytes; actions: drop
violated 0 packets, 0 bytes; actions: drop
Punt conformed 0000 bps, exceed 0000 bps, violate 0000 bps
<SNIP>

 “Clear control-plane” to clear the counters

ESP
QFP  Show platform software infrastructure punt-keepalive

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Troubleshooting Packet Drops
Good questions first

 What’s the problem ? Is the problem due to

packet drops ?
 Where/When packets are dropped ?
 Which packets are dropped ?
 How packets are dropped ?

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Packet Forwarding RP IOSd

Control Plane : Kernel

RIB – show ip route A.B.C.D …. Interconn.

AIB – show ip arp ….
LIB – show mpls ip ….
FIB – show ip cef A.B.C.D/nn ….
LFIB – show mpls forwarding-table ….
ESP Interconn.

Interconn.

Interconn.
Data Plane :
QFP
show platform hardware qfp active feature cef-mpls ?
adjacency CEF Adjacency Table Information
lb-alg Global load balance algorithm
prefix Show prefix
urpf uRPF subblock information Crypto

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 ASR1K Data Path IOSd

RP Kernel

Interconn.

SIP ESP Interconn. SIP

Slow Path
Interconn.

Interconn.
Interconn.
Interconn.
SPA SPA

SPA
SPA

Agg. QFP Agg.

Fast Path Crypto

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Ingress/Egress SIP/SPA
ESP show platform hardware slot x serdes statistics

show platform hardware slot x serdes status

Interconn.

IOCP
SPA
Agg.
show interface
show platform hardware slot x/y plim statistics
SPA … SPA show platform hardware slot x/y plim status

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 Enhanced Service Processor ESP RP0 RP1

Interconn.
QFP ( Quantum Forwarding Processor )
• Packet Forwarding Engine
• ACL/Firewall

Interconn.

Interconn.
• QOS SIP QFP SIP
• Crypto

Crypto

show platform hardware slot F0 serdes statistics

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
QFP Packet Handling
Packet Processing Elements Buffer Queuing Scheduling

DRAM … Pkt
PPEs & HW Assists Memory
PPEs & HW Assists
PPEs & HW Assists

IPM OPM
TCAM

Recycle
QFP

RP0 RP1 FP-Stby

CC0 CC1 CC2

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Commands for QFP Statistics
 show platform hardware qfp active datapath utilization
Overview
 show platform hardware qfp active statistics drop

 show platform hardware qfp active bqs 0 ipm interface statistics

 show platform hard qfp active bqs 0 ipm mapping IPM

 show platform hard qfp active bqs 0 ipm statistics channel all

 show platform hard qfp active interface <interface name> statistic

( Input interface )
QFP
 show platform hard qfp active interface <interface name> statistic
( Output interface )

 show platform hard qfp active bqs 0 opm mapping

OPM
 show platform hard qfp active bqs 0 opm statistics channel all
BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 Route Processor
show buffer
show ip traffic
show policy-map control-plane RP
IOSd

LSMPI (Linux Shared Memory Punt Interface)

• Shared IO Memory
Kernel
• Fast Punt / Inject packet path LSMPI
• Virtual interface
Interconn.
show platform software infrastructure lsmpi

show platform hardware slot RP serdes statistics ESP

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 Lsmpi_io pool

 Allocated and initialised during system boot process

 The lsmpi_io pool normally will show a very small amount free
 The amount of Free memory fluctuates a small amount

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Good Command for LSMPI
 show platform software infrastructure lsmpi
<SNIP>

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Three Policing on Punt Path
Control Plane Policers
Default: disabled
Enable by config

FP
RP
FECP IOSd

Kernel
QFP Punt Traffic
SPI4.2

Crypto subsys- Interconn.

assist tem

Interconn.

Per-Cause Punt-Policers
Always enabled but configurable
5
kpps
Global punt-policer
Fixed: 150kpps

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Monitor Punt/Inject Packets
show platform software infrastructure punt
FP
show platform software infrastructure inject
FECP

show platform software infrastructure lsmpi

QFP
show platform software punt-policer

SPI4.2
Crypto subsys-
assist tem

Interconn.

show platform hardware qfp active infra punt statistics type ?

global-drop Show aggregate drop statistics
inject-drop Show aggregate inject drop statistics
per-cause Show aggregate per cause punt statistics
punt-drop Show aggregate punt drop statistics

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Command examples for punt policers

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Troubleshooting High CPU
Good questions

 How high before we call it High CPU ?

 How long does High CPU last and how often?
 Any impact related to high CPU ?

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Impact of high CPU

 Slow response to telnet or console commands

 Routing protocol flappings
 Longer time for service recovery
 System crashes

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 ASR 1000 CPUs
FECP Forwarding RP CPU Chassis Mgr
Mgr. Punt
QFP IOSd Forwarding Mgr
(Active)
Client/Driver Inject
Chassis Mgr. Interface Mgr

Kernel Kernel

Enhanced Service Route Processor

Processor Control Messaging

IOCP Chassis Mgr.

SPA Interface
SPA
Driver Interface Mgr. Processor
SIP
Kernel

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
ASR1000 Platform CPU healthy

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 ASR1000 System Snapshot
show platform software process slot rp active monitor

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Troubleshooting High RP CPU
 IOSd shares RP CPU with other processes
show process cpu sorted | ex 0.00
CPU usage from IOSd’s view
show process cpu history
Not real RP CPU
show stacks <process id>

 Check RP CPU utilisation and Linux processes

show platform software status control-processor
show platform software process slot {r0|r1} monitor Real RP CPU
monitor platform software process

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Example - High IOSd CPU

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Example High CPU -
Problem description:
Customer experienced slow vty response - 5 seconds to see the
command outputs.

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Example High CPU -
 “Monitor platform software process { r0 |r1|f0|f1|0|1|2}”

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 Identifying Traffic Causing High RP CPU
RP CPU Control Plane Protection

In-band
Control
Data

Per-cause Punt Policers

In-box Packet captures ( TAC )
ESP Sniffer captures outside the box
QFP

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Troubleshooting FECP High CPU

show platform software status control-processor brief

show platform software process list fp active summary

show platform software process slot fp active monitor cycles

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Example – Busy FECP

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Monitor QFP load

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Troubleshooting Memory Issues
Symptoms for memory issues

 MALLOC failure messages

 Slow telnet sessions or unable to telnet
 Network service disruption
 Process dump or system reload

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
MALLOC Failure – Memory Exhaustion

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
MALLOC Failure – Memory Fragmentation

Requested a block of 16.8M bytes

IOSd only has largest block of 16.3M

IOSd still has 309M free memory

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Memory Leak
 Memory allocated but not freed after performing its function. So system
available memory would be reduced and eventually exhausted over time

 Mostly caused by on/off events or features, such as

Interface flapping
Session establishment/teardown
Routes add and remove
Config/unconfigure a feature
RP/ESP switchover, etc

 Memory Leak detection

Related to memory for specific feature
Establish reference point and check points
Sometimes take days to detect a leak

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
ASR1000 Platform Memory Overview

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
ASR1000 Memory Snapshot
monitor platform software process r0

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 RP Leak Detection

show platform software status control-processor brief

show platform software process list rp active
show platform software process slot rp active monitor cycles

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 ESP Leak Detection

show platform software status control-processor brief

show platform software process list fp active summary
show platform software process slot fp active monitor cycles

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Typical Commands for IOSd Memory Fragment
 show memory fragment
 show memory fragment detail
 show memory processor
 show memory summary
 show memory free
 show chunk / show chunk summary

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Typical Commands for IOSd Memory Leak
 show memory summary
 show process memory sorted
 show process memory <process id>
 show memory allocating-process total
 show memory debug leak //CPU Intensive Command

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Memory Allocation w/ Single IOSd

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Memory Allocation with Dual IOSds

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
IOSd Leak Detection

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Example – RP or IOSd ?

ASR1K

ASR1K

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 QFP Leak Detection

show platform hardware qfp active infrastructure exmem statistics

show platform hardware qfp active infrastructure exmem statistics user
show platform hardware qfp active tcam resource-manager usage

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Handle Unexpected Reload
Unexpected Router Reloads

 It is a software decision to crash, even though

it could be due to hardware error.
 Various components within ASR1000 that may
experience the issue:
IOSd, Linux Kernel, IOS-XE Process, SPA Driver,
QFP Microcode

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Unexpected Reload Symptoms

 Show version

 Show log

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Unexpected Reload Symptoms
 dir harddisk:core

 dir bootflash:

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Troubleshooting Unexpected Reloads
 Opening case with TAC, provide the following:
Is it the first occurrence?
What were the most recent changes?
What troubleshooting have you performed?
‘show platform’
‘show tech-support’
crashinfo / core dump file (if present)
Logs:
‘show logging’, syslog, tracelog, console log

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
System Monitoring
SNMP Monitoring
 IOSd CPU / Memory Utilisation
 RP CPU / Memory Utilisation
 ESP CPU / Memory Utilisation
 QFP Utilisation

 CISCO-PROCESS-MIB
 CISCO-ENTITY-QFP-MIB

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
SNMP Monitoring Memory Leak

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Application Visibility Control

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Application Visibility Control

 40 report types
 NBAR2 DPI 800 protocols
 Trend analysis
 Report export in different formats: PDF,
MS Excel, JPEG

BRKRST-2065: Application Visibility Control

- NBAR2, QoS, FNF and Insight Reporter

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Key Takeaways
Key Points

IOSd
RP
Kernel 3

Interconn.

Ingress SIP ESP Egress SIP

Interconn.
Interconn.

Interconn.
Interconn.

Interconn.

SPA SPA

SPA
SPA

Agg. QFP Agg.

Fast Path Crypto

1

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
 Is it enough ?

What’s next ?
BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Complete Your Online Session
Evaluation
Complete your session evaluation:
 Directly from your mobile device by visiting
www.ciscoliveaustralia.com/mobile and login
by entering your username and password

 Visit one of the Cisco Live internet

stations located throughout the venue

 Open a browser on your own computer

to access the Cisco Live onsite portal Don’t forget to activate your Cisco Live
Virtual account for access to all session
materials, communities, and on-demand and
live activities throughout the year. Activate your
account at any internet station or visit
www.ciscolivevirtual.com.
BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
 Q&A

BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
BRKSPG-3611 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 99