Nexus9000(Standalone)

Architecture and
Troubleshooting

Shridhar V. Dhodapkar-Technical Leader(Services)

BRKDCN-3101
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

Cisco Spark spaces will be cs.co/ciscolivebot#BRKDCN-3101

available until July 3, 2017.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Agenda
• Introduction
• Architecture-Brief
• Troubleshooting Toolkit
• Nexus 9000 Troubleshooting
• Common Link Layer Issues-L1
• L2/L3 Packet Forwarding
• Path of the Packet to CPU
• vPC Troubleshooting

• Nexus9000 Specific Limitation

and Goodies
Session Goal

• To provide an overall understanding of the Nexus 9000 switching

architecture, supervisor, fabric, and I/O module design, packet flows, and key
forwarding engine functions

• This session will introduce Troubleshooting tool Kits and troubleshooting

case scenarios

• This session will not examine NX-OS software architecture or other Nexus
platform architectures

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Related Sessions

• BRKARC-2222 - Cisco Nexus 9000 architecture

• BRKARC-3471 - Cisco NX-OS Software Architecture

• BRKDCN-3020- Network Analytics using N3K/N9K

• BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 Introduction-What is Nexus9000 Family

Nexus 9500 Series Switches Nexus 9300 Series Switches Nexus 9200 Series Switches

N9K-C9332PQ N9K-C9372PX N9K- N9K-C92160YC-X

Nexus9504/Nexus9508/Nexus9516
C9372TX N9K-C9396 Nexus 92304QC Nexus 9236C

Merchant Silicon/ Cisco ASICs Merchant ASICs Cisco ASICs

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Introduction to Different Forwarding ASIC
Merchant + Cisco Cisco ASICs-Cisco Cloud Scale

NFE(Trident-2) +ALE( Northstar)

ASE2(Lac) ASE3(Dav) LSE(Sug)

1st Generation 2nd Generation

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 Nexus 9500 Field Upgradeable Units (FRU)
• Nexus 9500 has the following modular components which can upgraded or
replaced in the field Nexus® 9508 Front View Nexus® 9508 Rear View

• Supervisor 3 Fan Tray

Line Cards
• Fabric Module Mid plane-Free
3 or 6 Fabric Module
• Line Card
• System Controller
Redundant System
• Fan Tray Controller

• Power Supply Redundant Supervisor

• The Supervisor, System controller ,Fabric Module and LC have OBFL (On-Board
Failure Logging) for failure analysis

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Nexus 9500 Platform FRU
Supervisor Module-What it is Role
• Redundant Half-width supervisor engine
• Common for 4-, 8-, and 16- slot chassis
• External Clock Input (PTP)
• Responsible for control-plane functions

System Controller-What it is Role

• Offload supervisor from internal device management tasks
• Central Point of Chassis Control
• EOBC Switch (Ethernet Out of Band Channel)
• EPC Switch (Ethernet Protocol Channel)
• Power Supplies via SMB (System Management Bus)
• Fan Trays

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 Nexus 9500 Platform Line Card
• I/O module with Merchant and FM6 FM5 FM4 FM3 FM2 FM1
Merchant+ ASIC
HG HG HG HG HG HG
MUX1 MUX4 MUX2 MUX5 MUX3 MUX6
• Have Various Forwarding Tables
• L2 Mac Table And L3 Host Table
• ACL and Buffers for Queuing
01 45 89 01 45 89
23
MN6Port
7 10 23
MN6 Port
7 10
ASIC Name Northstar 1 11
Northstar 2 11

MF Port MF Port
NFE=Network Forwarding Engine-Trident 2(T2) 0 3 6- 9- 0 9-
-
2
-
5
8 1
1 HG -
2
1
1
ALE=Application Leaf Engine-North Star(NS)
7 2 3 2 7 2
-Donner - Warpcor
- 1- 6- - 6-
5 0
T2 2
e 9
2
4
N9K-X9564PQ 5
T2 2
4

Note: Internal ports are called as Hi-Gig/HG ports

40G
QSFP
10G SFP+ Ports FP FP FP FP
F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F 49 50 51 52
F F F F F F F F F
P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P
PPPPPPP PP
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4
1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 36X100G Line card (N9K-X9736C-EX)
• Line card with Cisco ASICs has four LSE ASICs
• Next Generation of Fabric Module (N9K-C9508-FM-E ) are required
• EACH LSE on LC is in 8x100GE+16x50GE mode
• LSE on LC connected to all 4 FM using 16X50 GE links
• LSE is connected to the front-panel using 8x100GE links
• Stores Layer 2 MAC address , IPv4 host route and IPv4 LPM

FM0 FM1 FM2 FM3

16x50G
LSE-0 LSE-1 LSE-2 LSE-3
8x100G

QSFP Ports QSFP Ports QSFP Ports QSFP Ports

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Nexus 9500 Fabric Module
Fabrics Modules-Merchant Silicon

• Interconnect Line Card slots

• Installed at the rear of the chassis
• Leverages Broadcom Trident II ASICs
• Max 1.92 Tbps per line card slot (6 Fabric Cards)
• 960 Gbps per line card slot (3 Fabric Cards)
• All Fabric Cards are active and carry traffic
• Fan Tray requires Fabric Card to be present in even slot

Trident II Trident II
ASIC-NFE ASIC-NFE

32 x 40G 32 x 40G
Hi-Gig2 Hi-Gig2

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Nexus 9500 Next Generation Fabric Module
Next Generation Fabric Modules
ASE2 ASE2 ASE2 ASE2
ASIC ASIC ASIC ASIC
Maximum of 4 FM per chassis

• N9K-C9504-FM-E
• One ASE2 ASIC per FM
• 32x100G ports per FM
• N9K-C9508-FM-E 64x 50G
64x50G 64x50G 64x50G
• Two ASE2 ASICs per FM Hi-Gig2 Hi-Gig2 Hi-Gig2 Hi-Gig2

• 64x100G ports per FM

• N9K-C9516-FM-E
• Four ASE2 ASICs per FM Note: Only Stores IPv6 LPM/64 & IPv6 host routes
• 128x100G ports per FM

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
N9K-C9300 Series
Merchant-Plus Silicon
• Fixed Chassis

• Port QSFP+ Uplink Module

• 1 RU or 2RU or 3RU

• AC/DC Power Supply

Expansion Module
• Front-to-Back & Back-to-Front Airflow

• Latency: 1-2 usec

• Wire-Speed L2/L3 Forwarding

• Switch will not boot up without GEM

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
High Level Block Diagram-N9300
GEM 4x 40GE QSFP+ Uplinks

NorthStar

16GB Total
DIMM2
PCIe

DDR3
Northstar ASIC 1 CPU
2C 1.5GHz
Egress Ingress
• The last 2/3 numbers stand for
12 x 40G
(12+12)x12 (12+12)x12
Hi-Gig2 total bandwidth in Gigabits
1000BaseT • 93128 – 128G (96 x 10G + 8 x 40G)
Trident II Mgmt Port
ASIC • 9396 – 96G (48 x 10G + 12 x 40G)
BRCM Trident2
2 USB
Ports • 9372 – 72G ( 48 x 10G + 8 x 40 G)
12 x 40G
48 10G
Ethernet x 12 40G eUSB
Boot Flash
Network Interfaces 12C

Front Panel 48x 1GE/10GE Ports

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 Nexus9500-T2-NS Unicast Packet Flow
Fabric Module
Fabric Module
Performs L3 LPM
L3 LPM Lookup & Forwarding
lookup and resolves
Egress port and
Ingress Line Card Egress Line Card
next-hop
ALE-NS EoQ ALE-NS EoQ
Additional buffer is
Classify traffic T2-NFE OOBFC T2-NFE OOBFC available for
Ingress Signaling Ingress
based on 802.1q Signaling extended out put
Accounting & Accounting &
COS, IP Pres, Output Q Ques EoQ
Policing Policing Output Q
DSCP &ACL & Shaping
Remark if needed Traffic Traffic & Shaping
Classification Classification
E-ACL
& Remarking & Remarking E-ACL Class-based output
L2/L3 Lookup in queues. Support 6
I-ACL I-ACL classes including
MAC Table and IP Packet Packet
Host Table L2/L3 Modification L2/L3 Modification control traffic class
Lookup & Lookup &
forwarding forwarding
Parse the first 128 Egress Line card
Byte and extract Parser sends packet to
Parser
header info egress port based
on DMOD/DPORT
Network Interface Network Interface

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 N9K-C9200 Series
Cisco ASICs
Nexus 9236C
• Fixed Chassis-Cisco ASIC

• Port QSFP+ Uplink Module

• 1 RU or 2RU Nexus 9272Q

• 16GB DIMM memory/2MB NVRAM

• 64GB -SSD Nexus 92160-YC-X

Q: Native 40-Gbps front-panel ports
• Port-Side intake and exhaust Y: Native 25-Gbps front-panel ports
C: Native 100-Gbps front-panel ports
• Wire-Speed L2/L3 Forwarding X (after the hyphen): Cisco NetFlow and data analytics capabilities

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Nexus9000 ASIC Components- IO Components

ASE2/ASE3/LSE • High Speed SerDes

• Determine Bandwidth capacity

Slice Component
• Multiprotocol MAC
IO Component Input Slice
• Packet Parser
Output Slice
• Forwarding Controller
Slice Component Slice 1 Slice 2 Slice 3 Input Slice
• Input and Output Packet buffering
Output Slice
• Buffer Accounting
Central Broadcast Broadcast Network
Global Component
Statistic Network
• Output queuing and Scheduling
/ Central Statistic
• Output Rewrite
Input Slice
Slice Component Slice 4 Slice 5 Slice 6 Global Component
Output Slice
• PCIe Controller register and eDMA
Broadcast Network access
IO Component
• Broadcast Network to connect all
Between Multiple Slice slice
• Collects packet statistics
• PLL to generate core and MAC
clocks
BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Packet Flow-Cisco ASICs
Ingress Processing

Parse Ingress Ingress Forwarding Pause Accounting

Packet Layer 2 Layer 3 ACL Traffic Result And
Lookup Lookup Processing Buffer Flow Control
Header Classification Generation

Input Forwarding Input Data Path

IO/MAC
Controller Controller Broadcast
Network/Central
Slice Statistics
Output Forwarding Output Data Path
IO/MAC Controller Controller

Packet
Egress Egress
Packet Multicast Multicast Queueing
ACL Buffer
Rewrite Fan-out Replication And
Processing Accounting
Shaping

Egress Processing
BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 Unified Forwarding Table
• System uses combination of dedicated TCAM table space and shared hash table
memory known as Unified Forwarding Table (UFT)
• UFT stores Layer-2 and Layer-3 forwarding information
• Layer 3 Forwarding Architecture Categories
• Modular Multi-Chip-LC and FM contribute in making Forwarding decision-Modular
Chassis EOR
• Fixed Single Chip-Single Chip make forwarding decision for whole system-TOR

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
UFT On N9500 with T2/NFE

Fabric 1 Fabric 6
Mode L2 L3 Host LPM
… Fabric Modules
NFE NFE NFE NFE
0 288K 16K 16K Mode 4

1 224K 56K 16K

2 160K 90K 16K

3 98K 122K 16K Line Card/IO

4 32K 16K 128K NFE NFE NFE module
Mode-3

36-port 40 Gbps QSFP+

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 Routing Mode for Nexus9500-LSE-ASE2
• Following prefixes are stored on FM Fabric 1 Fabric 6
ASE ASE … ASE ASE
Fabric Modules
• IPv6 /64 prefix length – 176K 2 2 2 2

• IPv6 ND learnt /128 – 32K

• IPv6 prefix not /64 – 3.9K
• Line Card with LSE Stores IPv4 Prefixes

LSE LSE LSE

Line Card/IO
module

36-port 40 Gbps QSFP+

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 Routing Mode on N9200/N9300
Modes for Cisco Nexus 9300 Series Switches
LPM Routing Mode Broadcom T2 CLI Command
Mode
Default system routing mode 3
ALPM Routing mode 4 System routing max-mode l3

Modes for Cisco Nexus 9200 and 9300-EX Series-CicoASIC’s

Mode L2 L3 Host LPM

Default 96K 96K 14K

Note:Max Mcast
Route Entries
32 K 32K 64K Limit=0 other than
LPM Heavy
for Default Mode
Dual Stack Host 64k 160K 16K
Scale

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Verifying System Routing Mode

N93K#show system routing mode

Configured System Routing Mode: Hierarchical
Applied System Routing Mode: Hierarchical (Default)
N93K#show hardware internal forwarding table utilization module 1
Max Host Route Entries (shared v4/v6): 124928
Max LPM Table Entries : 16384

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 Buffer And Queuing –T2 & NS
T2 Buffer Carving With NS on Board
• T2 has 12 Mbytes of Buffer Control
Shared by all Ports for all Traffic
Shared
OOBFC
• North Star has 40 Mbytes of Buffer GEM 4x 40GE QSFP+ Uplinks Buffer
12 MB
• T2 Shared buffer divided into Control, 10 MB NorthStar 20 MB
Buffer ASIC Buffer
Default
default and OOBFC Service pool
10 MB Buffer
• North Star buffer divided in to Control 12 x 40G
Hi-Gig2
Span and Default Service pool North star Buffer Carving
• NS 10 MB used with Buffer Boost Trident II
Control
ASIC
• Raw Memory divided in 208 Byte
12 x 40G SPAN
Shared
called cells Ethernet
Front Panel 48x 1GE/10GE Ports Buffer
• OOBFC on T2 is out of band flow Default

control only for Unicast

LC with only with T2 Buffer divided in t Control and Default pool

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Buffer Boost Function With T2 and NS
Fabric Module
• Buffer boost is function which allow T2 to use extra
buffer of NS
• When Buffer boost is enabled on a port , T2 Local ALE-NS
switch traffic is Sent to NS for extra buffer space- 10 MB 20 MB
Buffer 10 MB Buffer
• When Buffer boost is disabled on a port, T2 local traffic Buffer
to this port remains local on this NFE
• Buffer Boost is enabled by default and can be disabled
NFE 12 MB Buffer Shared by all
on a per port basis T2 ports

Network Interface

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Buffers on Cisco ASICs ASE2/ASE3/LSE
• The raw memory is organized into 208-byte cells ASIC
Buffer Per
Slice
Total Buffer

• By Default Class base Egress Queueing

ASE2 5.1 MB 30.6 MB
• Most buffer Cells allocated to Egress Queue
LSE 18.7 MB 37.4 MB
• With PFC buffer cells allocated to ingress queue
• Software can partition the buffer into a maximum of four pool groups ASE3 10.2MB 20.4MB

30.6 MB total Buffer

37.4 MB total Buffer
20.4 MB total Buffer
5.1 MB 24 K
18.7 MB
LSE 18.8 MB ASE2 5.1 MB 24 K
10.2 MB 10.2 MB
Cell/Per Slice Cell/Per Slice
48 K Cell/Per Slice 48 K Cell/Per Slice
Slice 1 Slice 2 Slice 6 48 K Cell/Per Slice ASE3 48 K Cell/Per Slice
Slice 1 Slice 2 208 B Cell Slice 1
208 B Cell 208 B Cell Slice 2
208 B Cell 208 B Cell 208 B Cell 208 B Cell
208 B Cell 208 B Cell 208 B Cell 208 B Cell
208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell
208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell
208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell
208 B Cell 208 B Cell 208 B Cell
208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell 208 B Cell
208 B Cell 208 B Cell 208 B Cell

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 ACL TCAM TABLE –T2
Characteristic

• Ingress ACL: 4K TCAM entries - 4x 512 banks + 8x 256 banks

• Egress ACL: 1K TCAM entries - 4x 256 banks
• Each ACL type needs its own dedicated bank/banks
• IPv4, IPv6 or MAC each needs dedicated bank/banks
• MAC-ACL IPv6 & any QOS needs double-width entries, which means needs at least 2 banks
• VACL is programmed symmetrically in both egress and ingress ACL

Interface
Ingress ACL Egress ACL
Type

SVI TCAM Shared TCAM Not Shared

L3 TCAM Shared TCAM Shared

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
TCAM Carving for Nexus9000
TCAM Region- Size per Ingress TCAM Region- Size per Ingress
N9500 Region N9300 Region
IPV4 RACL 1536 3x512
IPV4 PACL 512 1x512

IPV4 L3 QOS 256 1x256 IPV4 VACL 512 1x512

IPV4 RACL 512 1x512
Ingress System 256 1x256
IPV4 Port QOS 256 1x256
SPAN 256 1x256 Ingress System 256 1x256

Ingress CoPP ERROR:

256 Aggregate
1x256 TCAM region configuration exceeded the available
SPAN 256 1x256
show system internal access-list global - Verifies the current hardware
Redirect
configuration
512 1x512
Redirect Nexus9300(config)#
256 1x256 hardware access-list tcam region racl 0
vPC
Warning: Please save config and reload the system for the configuration 256effect 1x256
to take
Convergence
vPC Convergence 512 1x512
Egress
Egress
Egress IPV4 768 3x256
Egress IPV4 768 3x256 RACL
RACL
Egress System 256 1x256
Egress System 256 1x256

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 TCAM Carving For LSE/ASE2/ASE3
TCAM
TCAM Region-N9500 Size Per Region

IPV4 RACL 1792

IPv4 L2 QOS 256 Ingress 3582
Ingress L2 QOS 256
Ingress
Ingress L3/VLAN QOS 512
Ingress SUP 512
COPP 512
Ingress L2 SPAN ACL 256
Ingress L2 VLAN SPAN 512 Default entry 2
ACL
SPAN 512
Egress 1790
Egress RACL 1792
Egress
COPP 256
Egress Sup 256
Default entry 2
BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
ACL – ASE2/ASE3 /LSE
• Ingress ACL TCAM-Single Width
There are two banks of size 2K (consisting of two tcams of size 1K) for a total of 4K
(4096) Entries
• Egress ACL TCAM-Single Width
There are two banks of size 1K (consisting of two tcams of size 0.5K) for a total of
2K (2048) Entries
• V4/V6 of given feature goes in same region
• Consistency Checker will run against all Slices
• QoS TCAM lite regions are not supported

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 Agenda
• Introduction
• Architecture-Brief
• Troubleshooting Toolkit
• Nexus 9000 Troubleshooting
• Common Link Layer Issues-L1
• L2/L3 Packet Forwarding
• Path of the Packet to CPU
• vPC Troubleshooting

• Nexus9000 Specific Limitation

and Goodies
When To Use Ethanalyzer Or TCP Dump
• To Analyze the traffic sent and received by CPU
Netstack
• It uses wireshark’s code (an open source software)
• Troubleshooting High CPU SUP
Pseudo Inband

• Troubleshoot Control Plane issues Ex. OSPF, PIM,

STP Flap NIC-ETH2

• Tcpdump command works on most flavors of Linux

operating system
• More info at http://www.tcpdump.org/

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Elam-Embedded Logic Analyzer
• Elam Allows to capture single packet based on Trigger
• Triggers are configured using Packet information
• Only Supported on LSE/ASE2/ASE3/ALE
• Use with TAC Supervision
• Help to Answer some of the Question
• How did the Packet look like ?
• How was the packet rewritten based on forwarding Decision
• Was the packet correctly forwarded or was Dropped ?

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Packet Tracer And Flex Counters–T2 Only FM Mod OR NS

• Packet Tracer
• Helps to Trace the packet inside Switch Trident II
ASIC
• Only packets in the direction of the flow are traced
• Two ACLs are installed for each filter on each Line card Network
• One ACL for Front Panel Port Group Interfaces
• Second ACL for Traffic exiting Fabric Module and ingressing Line card
Nexus9508 with
N9K-X9564TX
• Flex counters
• Flex counters used to count Next hop Adjacency stats Eth6/1 Eth6/52

• One Stat Counter per adjacency

• One can attach Stats to multiple Adjacency at same time 192.1.1.1/3
0 192.1.1.2/30

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Consistency Checkers-Summary
• Show consistency-checker stp-state vlan
• Show consistency-checker link-state
• Show consistency-checker membership vlan
• Show consistency-checker membership port-channels
• Show consistency-checker membership port-channels
• Show consistency-checker l2
• Show consistency-checker l3
• Show consistency-checker forwarding ipv4 unicast
• Show consistency-checker Copp
• Show consistency-checker fex-interfaces fex <101-199>

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
SPAN & ERSPAN
• Switch Port Analyzer”
• Provides efficient, high-performance traffic monitoring service
• Duplicates network traffic to one or more monitor interfaces Sup-eth
• Types Of SPAN
• Local SPAN e6/1 e6/2
• Encapsulated Remote SPAN(ERSPAN) Local SPAN
• Applications:
• Troubleshooting connectivity issues Layer 3
• Base lining network utilization/performance L3
• Detecting anomalous traffic flows
• On Nexus9000 Span Traffic uses dedicated queue e6/1 e6/2
• Queue carrying SPAN traffic has low Priority over other queue’s ERSPAN
during congestion

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Troubleshooting Toolkit-Summary
• To analyze the traffic sent and received by CPU
• Ethanalyzer Or TCP Dump
• To capture single packet switched in hardware
• ELAM- Available on Cisco ASICs LSE/ASE2/ASE3 and ALE
• To trace the packet inside switch on T2 base hardware
• Packet Tracer
• To count Next Hop Adjacency stats on T2 base hardware
• Flex Counter
• To duplicate network traffic to one or more interfaces
• SPAN and ERSPAN
• To verify hardware programming
• Consistency checkers
BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 Agenda
• Introduction
• Architecture-Brief
• Troubleshooting Toolkit
• Nexus 9000 Troubleshooting
• Common Link Layer Issues-L1
• L2/L3 Packet Forwarding
• Path of the Packet to CPU
• vPC Troubleshooting

• Nexus9000 Specific Limitation

and Goodies
General Recommendation For New and Existing
Deployments
• Software Recommendation
Platform Recommended Version

Cisco Nexus 9000 Series Switches NX-OS 7.0(3)I4(6)*

Cisco NX-OS 7.0(3)I2(x) is long-lived release train for the CiscoNexus9000 (Merchant-Plus) Hardware
*Proposed long-lived release is 7.0(3)I4(x)
Cisco Nexus 9000 Series NX-OS Recomanded Release Notes

• Verified Scale Limits for different features and protocol for each Relapse
Cisco Nexus 9000 Series NX-OS Verified Scalability Guide

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Topology Used For Troubleshooting
L3-Links

N9500 N9500 L2-Links

N9396PX-1 Eth1/48 N9396PX-2 N92160YC-X Eth1/48 Eth1/48 N92160YC-X

Eth1/48
Eth1/31 Eth1/31 Eth1/31 Eth1/31
Eth1/32 Eth1/32 Eth1/32 Eth1/32
Eth1/1 Eth1/1
Eth1/1 Eth1/1

vPC vPC
Eth1/1 Eth1/2 Eth1/1 Eth1/2

IP=10.10.201.20/24
IP=10.10.202.20/24 IP=10.10.101.20/24 IP=10.10.102.20/24
Mac=0000.c003.0102
Mac=0000.c003.060c Mac=0000.0090.254e Mac=0000.e3db.0076
VLAN0201
VLAN0202 VLAN0101 VLAN0102

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Forwarding ASIC Terminology

ASE2(Lacrosse) ASE3(Davos) LSE(Sugarbowl)

ASIC Manufacturer Hardware Name Marketing Name Troubleshooting command

Cisco North Star(NS) ALE module-1# sh hardware internal ns

Cisco Sugar Bowl LSE Module-1# show hardware internal tah l3
module-1# debug hardware internal sug
Cisco Lacrosse ASE2 Module-1# show hardware internal tah l3
module-1# debug hardware internal lac
Cisco Davos ASE3 Module-1#show hardware internal tah l3
module-1# debug hardware internal dav
Broadcom T2 NFE N9K#bcm-shell mod 1 “0:ps”

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Which Forwarding ASIC Present on Hardware ?
module-1# show hardware internal dev-version
-------------------------------------------------------------------
Name InstanceNum Version Date
-------------------------------------------------------------------
Sugarbowl 0 0x02 N9K-X9732C-EX
Sugarbowl 1 0x02
Sugarbowl 2 0x02
Sugarbowl 3 0x02
IO FPGA ASIC Name 0 0x06 20160309
MI FPGA 0 0x06 20160314
CPLD1 0 0x0e

module-21# show hardware internal dev-version

-------------------------------------------------------------------
Name InstanceNum Version Date N9K-C9504/8-FM-E
-------------------------------------------------------------------
Lacrosse 0 0x01
Lacrosse 1 0x01
IO FPGA ASIC Name 0 0x12
20160505

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Understanding T2 Interface –Xe0/hg
N9K# bcm-shell mod 1 "show unit"
Unit 0 chip BCM56852_A2 (current) hg0 hg11 hg0 hg11
Unit 1 chip BCM56852_A2
Internal
N9K#bcm-shell mod 1 “0:ps” T2
Instance 0
ports T2
Instance 1
ena/ speed/ link auto STP lrn inter max loop
Front Port
port link duplex scan neg? state pause discrd opsF Fface
QSPFF F
frame
F
back F F F F FQSPF
F F F F F F
FPFP FP FP FPFP FP FP
P P P P P P P P P PPorts P P P P P P
01 02 04 Ports
06 09 10 12 19
hg0 up 42G FD HW No Forward None FA XGMII 16360 03 05 07 08 11 13 14 15 16 17 18 20 21 22 23 24

hg2 up 42G FD HW No Forward None FA XGMII 16360

--------------------------------Snip----------------------------------
Xe0 Xe11 Xe0 Xe11

Hg11 up 42G FD HW No Forward None

Eth1/1 FA XGMII
Eth1/12 16360
Eth1/13 Eth1/24
Xe0 !ena 40G FD HW No Disable None FA XGMII 1582
xe1 up 40G FD HW No Disable None FA XGMII 1582
--------------------------------Snip----------------------------------
Xe11 !ena 40G FD HW No Disable None FA XGMII 1582

Hg=Internal Ports
Xe=Front Panel Port
BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Understanding T2-NS interface Pinning
module-1# show hardware internal ns mac pinning
UNIT 0 Front-ports-40Gig

===
--------------------------------------------------------------------------------
Mapping for ASIC instance 0: MACN-ports
40 Gig Ports
-------------------------------------------------------------------------------- NS/ALE on GEM
Front-port MACN-port MACF-port HG-port MACF-ports

1 10 1 10
2 11 0 11 HG-ports
3 8 3 8 MACN=40 Gig port on GEM T2
4 9 2 9 Instance 0
Hg=Internal Ports
---------------------------Snip---------------------------------------- Front Ports on
9 2 9 2 Base Board
10 3 8 3 Xe=Front Panel Port 1/10 Gig QSPF
11 0 11 0 FP FP FP FP FPPorts
MACN/MACF/HG are “0” base
FP FP FP FP FP FP FP
01 02 03 04 05 06 07 08 09 10 11 12
12 1 10 1
module-1#show hardware internal dev-version Front-ports- 1/10Gig

-------------------------------------------------------------------
Name InstanceNum Version Date Xe0 Xe11
-------------------------------------------------------------------
Forwarding ASIC 0 0x03 Eth1/1 Eth1/12
NorthStar 0 0x02

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Layer -1 Issues- Link Not “UP” On T2
C9396PX-1# show interface Ethernet 1/1
Ethernet1/1 is down
admin state is up, Dedicated Interface
Last link flapped 00:00:01
module-1# show hardware internal bcm-usd event-history front-port 1
2)Event:E_STRING, length:111, at 417271 usecs after Wed May 3 23:49:42 2017
bcm_usd_notif_link_down(248): [unit=0 nxosport=1 bcmport=13] sending LINKDOWN to port client
reason 0x40e50005
module-1# show system error-id 0x40e50005
Error Facility: port_client
Error Description: Autonegotiation failed

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Layer -1 Issues- Link Not “UP” On Cisco ASICs
module-1# show hardware internal tah event-history front-port 49
2)Event:E_STRING, length:111, at 417271 usecs after Wed May 3 23:55:42 2017
tahusd_port_sm(556): [inst=0 nxosport=144 fp_port:145] Took too long to bring up link. Shutting down and
restart..
tahusd_linkscan_thread(171): [inst=0 nxosport=144 fp_port:145] Linkscan scheduled for next in 0 ms
Possibility of Unsupported cable or Transceiver

Cisco 40-Gigabit Ethernet Transceiver Modules Compatibility Matrix

Cisco 100-Gigabit Transceiver Modules Compatibility Matrix

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
L2 Mac And Vlan Table Verification –T2
N9K# sh mac address-table dynamic vlan 100
Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay Eth6/1
MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - Mac=547f:ee1c.06fc
True, (F) - False
interface Ethernet6/1
VLAN MAC Address Type age Secure NTFY Ports switchport
* 100 547f.ee1c.06fc dynamic 0 F F Eth6/1 switchport access vlan 100
no shutdown
N9K# bcm-shell mod 6 " l2 show" | in Hit
mac=54:7f:ee:1c:06:fc vlan=100 GPORT=0x800800d modid=16 port=13/xe0 Hit

N9K# bcm-shell mod 6 "vlan show 100”

vlan 100 ports xe0,hg ....... untagged xe0

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
L2 Mac Table Verification on Cisco ASICs
module-1# show hardware internal tah sdk-l2 entries
L2 Entries Vlan No.

Instance : 0
DstNivIdx: 1538 DstIdx: 3 MacAddress: 54:7f:ee:1c:06:fc BD: 888 FPX:5 Tile:0 Index:218 Ocam:0 TcamIndex:0
OwnerFlags:L2/3(0x3) NH:0x90007 Age:0

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Spanning Tree Verification-NFE
N9K# sh spanning-tree interface ethernet 6/1
Vlan Role Sts Cost Prio.Nbr Type
interface Ethernet6/1
VLAN0100 Desg FWD 4 128.1537 P2p switchport
N9K# bcm-shell mod 6 "dump vlan 100” switchport access vlan 100
no shutdown
VLAN.ipipe0[100]: <VP_GROUP_BITMAP=0x00000……STG=0X67
FID_ID=0x64
N9K# Dec 0x67=103
N9K# Dec 0x64=100
STG= STP Group ID Eth6/1
FID_ID=Vlan ID.
Mac=547f:ee1c.06fc
N9K# bcm-shell mod 6 "stg stp 103”
STG 103:
Block: xe1-xe47
Forward: xe0,hg

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
L3 Unicast Troubleshooting Flow
Next-Hop
Check the routing table Show ip route [ipv4] [<prefix>]

ARP/MAC
Show ip arp [ipv4]
show ip adjacency (Ipv4]
Check the ARP Table
show forwarding adjacency platform [ipv4] module
<mod>
Checking Route on
RIB And FIB.
Check Forwarding Route show forwarding [ipv4] route module <mod>

HW Programming
On LC/FM
bcm-shell mod 22 "l3 defip show”
Use BCM commands show hardware internal tah l3

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 Verifying Flow seen by Switch-Packet Tracer !!!!
C9396PX-1#test packet-tracer dst-ip 10.10.101.20 src-ip 10.10.201.20 detail-fp
C9396PX-1#test packet-tracer start
C9396PX-1#test packet-tracer show
Module 1:
Filter 1 installed: src-ip 10.10.201.20 dst-ip 10.10.101.20 detail-fp
ASIC instance 0:
Entry 0: id = 7425, count = 0, active, fp, port 0 N9396PX-1 Eth1/48 Eth1/48 N9396PX-2
Entry 1: id = 7426, count = 77593, active, fp, port 13 Eth1/31 Eth1/31
Eth1/32 Eth1/32
Eth1/1 Eth1/1
C9396PX-1# show Interface hardware-mappings | in 13 vPC
--------------------------------------------------------
Name Ifindex Smod Unit HPort FPort NPort Vport
--------------------------------------------------------
Eth1/1 1a000000 1 0 13 255 0 -1 IP=10.10.201.20/24 IP=10.10.202.20/24

Mac=0000.c003.0102 Mac=0000.c003.060c

VLAN0201 VLAN0202

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Router MAC Programming Check
• Router Mac address must be programmed in Hardware
N92K-1# show hsrp interface vlan 201 | grep Default
Virtual mac address is 0000.0c9f:f0c9 (Default MAC)

C9396PX-1#bcm-shell mod 1 " d chg my_station_tcam" | grep f0c9

MY_STATION_TCAM.ipipe0[3]:<VLAN_ID_MASK=0xfff,VLAN_ID=0xc9,VALID=1,MASK=0xfffffffffffffff,MAC
_ADDR_MASK=0xffffffffffff,MAC_ADDR=0xc9ff0c9,KEY=0xc900000c9ff0c9,IPV6_TERMINATION_ALLOW
ED=1,IPV4_TERMINATION_ALLOWED=1,DATA=0x38,ARP_RARP_TERMINATION_ALLOWED=1>

module-1# show hardware internal tah rmac

Instance : 0
==================================== N92K-1# show hsrp interface vlan 888 | grep Default
Mac-Address Vlan Flag Virtual mac address is 0000.0c9f.f378 (Default MAC)
------------------------------------
00:00:0c:9f:f3:78 888 VRMAC

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
BRKDCN-3101
 Route Validation
C9396PX-1# show ip route 10.10.101.20
IP Route Table for VRF "default”
10.10.101.0/24, ubest/mbest: 2/0
*via 192.1.1.1, Eth2/7, [110/42], 01:25:46, ospf-10, intra
*via 192.1.1.17, Eth2/8, [110/42], 01:25:46, ospf-10, intra
C9396PX-1# show forwarding route 10.10.101.20 module 1
IPv4 routes for table default/base
Prefix | Next-hop Interface | Labels

10.10.101.0/24 192.1.1.1 Ethernet2/7 Multi-Path

192.1.1.17 Ethernet2/8

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
 Route Validation -Hardware
C9396PX-1# bcm-shell mod 1 "l3 defip show" | grep 10.10.101.0
# VRF Net addr Next Hop Mac INTF MODID PORT PRIO CLASS HIT VLAN
2689 1 10.10.101.0/24 00:00:00:00:00:00 200257 0 0 0 0 y (ECMP)
C9396PX-1# bcm-shell mod 1 "l3 multipath show”
C9396PX-1# show Interface hardware-mappings | in 2/7|2/8
Multipath Egress Object 200257 --------------------------------------------------------
Name Ifindex Smod Unit HPort FPort NPort Vport
--------------------------------------------------------
Interfaces: 100012 100013 Eth2/7 1a006c00
Eth2/8 1a006e00
1 0 5
1 0 6
4
5
54 -1
55 -1

C9396PX-1#bcm-shell mod 1 "l3 egress show" | in 100012|100013

Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop
100012 78:da:6e:71:9a:3f 64 4160 5 1 -1 no no
100013 00:3a:99:fc:dd:7f 65 4161 6 1 -1 no no

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 Hitting Right adjacency ?-FLEX Counters !!!
C9396PX-1#sh routing hash 10.10.201.20 10.10.101.20 mod 1
Hashing to path *192.1.1.17

Out Interface: Eth2/8

For route:

10.10.101.0/24, ubest/mbest: 2/0

*via 192.1.1.1, Eth2/7, [110/42], 02:15:20, ospf-10, intra

*via 192.1.1.17, Eth2/8, [110/42], 02:15:20, ospf-10, intra

C9396PX-1#test hardware internal adjacency statistics nexthop ipv4 192.1.1.17 interface et2/8 enable

C9396PX-1#test hardware internal adjacency statistics nexthop ipv4 192.1.1.17 interface et2/8 show

adjacency counters for nhip 192.1.1.17 if Ethernet2/8:

Ucast: Packets 2257379 Bytes 173366216

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Packet Egressing on GEM N9396PX-1
Eth1/48 Eth1/48
N9396PX-2

Ports-ELAM !!
Eth1/31 Eth1/31
Eth1/32 Eth1/32
Eth1/1 Eth1/1

vPC

C9396PX-1# attach mod 1

module-6# debug platform internal ns elam asic 0
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)#set outer ipv4 dst_ip 10.10.101.20 src_ip 10.10.201.20
module-6(NS-elam-insel3)# start GBL_C++: [INFO] ce_da: 003A99FCDD7F
module-6(NS-elam-insel3)# status GBL_C++: [INFO] ce_sa: 7C69F6DFC227
module-6(NS-elam-insel3)# report GBL_C++ [INFO] hg2_srcmod: 0E
GBL_C++ [INFO] hg2_srcpid: 0D Information
is in Hex
GBL_C++ [INFO] hg2_dstmod: 33 Convert to
GBL_C++ [INFO] hg2_dstpid: 06 Dec.

GBL_C++ [INFO] ip_da: 000000000000A0A6514

GBL_C++ [INFO] ip_sa: 000000000000A0AC914

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 Verify HW-Programming on Spine-N9500
This is not /32 host Route.
N95K-B#show ip route 10.10.101.20
Packet forwarding decision
IP Route Table for VRF "default” responsibility is of the Fabric Module
10.10.101.0/24, ubest/mbest: 2/0
*via 192.1.1.22, Eth1/36, [110/41], 05:10:38, ospf-10, intra
*via 192.1.1.30, Eth4/36, [110/41], 05:10:38, ospf-10, intra
N95K-B# show forwarding route 10.10.101.20 module 21
ALL FM will be programmed
IPv4 routes for table default/base
with this Route
Prefix | Next-hop Interface | Labels
10.10.101.0/24 192.1.1.22 Ethernet1/36
192.1.1.30 Ethernet4/36
N95K-B# show hardware internal forwarding adjacency statistics default-route mod 1
Module:1 Unit:1
Traffic matched adjacency for default route (destined to FM):
Unicast: Packets 51771793 Bytes 51809193068 Install a default route 0/0 on all LC to
forward packets to FM .
FM performs LPM lookup and make
forwarding decision
Follow same steps now on FM to troubleshoot HW programming as earlier .

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Troubleshoot Fabric Interface Drops –N9500
F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F
P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P
25 26 27 28 29 30 31 32 33 34 35 36 13 14 15 16 17 18 19 20 21 22 23 24 01 02 03 04 05 06 07 08 09 10 11 12
QSPF Ports QSPF Ports QSPF Ports

T2 T2 T2
Instance 0 Instance 1 Instance 0 Line Card

HG Ports HG Ports HG Ports

FM1 FM2 FM3 FM4 FM5 FM6 Fabric Modules

HG Ports HG Ports HG Ports

T2 T2 T2
Instance 0 Instance 1 Instance 2
Line Card

F F F F FQSPF F F F F F F F F F F F FQSPF F F F F F F F F F F F FQSPF F F F F F F F

P P P P PPorts P P P P P P P P P P P PPorts P P P P P P P P P P P PPorts P P P P P P P
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Fabric Port Drops and Link Status
N95K-B# show hardware internal fabric interface asic counters mod 1
Counters for Fabric Ports:
FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
RxDrops TxDrops RxDrops TxDrops Drops Drops
0 / 1 / HG0 0 0 0 10 99 0
0 / 2 / HG1 0 0 10 0 0 0

N95K-B# show hardware internal fabric interface asic counters mod 21

Counters for Fabric Ports:
FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
RxDrops TxDrops RxDrops TxDrops Drops Drops
0 / 1 / HG0 0 0 0 0 0 0
1 / 1 / HG0 0 0 10 0 0 0

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
 Check for Drops/Errors-Fabric Module
Nexus9500# show system internal fabric connectivity mod 1 | grep 22 Identify HG Port on LC and FM
LC-Slot LC-Unit LC-iEthLink MUX FM-Slot FM-Unit FM-iEthLink

1 0 iEth03 - 22 0 iEth09

1 0 iEth05 - 22 1 iEth4
Nexus9500#show hardware internal fabric interface asic counters mod 22
Important Counters/Drops
--------------- --------- --------- --------- --------- --------- ------------------------------
Interface Name Drop Reasons for the Interface, See below output for detail if any
--------------- --------- --------- --------- --------- --------- ----------------------------
|9|9|9|9|9|9|8|8|8|8|8|8|8|8|8|8|7|7|7|7|7|7|7|7|7|7|6|6|6|6|6|6|6|6|6|6|5|5|5|5|5|5|5|5|5|5|4|4|4|4|4|4|3|2|2|2|2|2|1|1|1|1|1|1|1|1|1|0|0|0|0|0|0|0|0
|5|4|3|2|1|0|9|8|7|6|5|4|3|2|1|0|9|8|7|6|5|4|3|2|1|0|9|8|7|6|5|4|3|2|1|0|9|8|7|6|5|4|3|2|1|0|6|5|4|3|1|0|8|6|5|2|1|0|9|8|6|5|4|3|2|1|0|9|8|7|6|5|4|3|1

iEth3 |.|.|.|.|.|.|.|.|.|.|X|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.| X|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.|.

--------------- --------- --------- --------- --------- --------- ------------------------

Drop Conditions Drop Reason
67 : TAHOE Ingress DROP_ACL_DROP
------------ -----------------------------------------------------------------------------
Nexus9500# clear hardware internal fabric interface asic counters mod 22

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Fabric Troubleshooting commands
Fabric Module Slot-21
N95K-B#show system internal fabric connectivity mod 1
HiGIG Link-info Linecard slot:1 T2 T2
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink #0 #1
HG00 HG00
1 0 HG00 - 21 0 HG00

1 0 HG01 - 21 1 HG00

Slot 21 to Slot 26 assign to Fabric Module

N95K-B#show system internal fabric connectivity mod 21
T2 T2 T2
HiGIG Link-info Fabriccard slot:21 #0 #1 #2

FM-Slot FM-Unit FM-HGLink LC-Slot LC-Unit LC-HGLink MUX

21 0 HG00 1 0 HG00
Line Card Slot-1
21 1 HG00 1 0 HG01

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Troubleshooting On TOR- Cisco ASIC
Route Validation
N92160-1# show ip route 10.10.101.20
IP Route Table for VRF "default”
10.10.101.20/32, ubest/mbest: 1/0, attached
*via 10.10.101.20, Vlan101, [250/0], 2w3d, am
N92160YC-X Eth1/48 Eth1/48 N92160YC-X
N92160-1# show ip arp Eth1/31 Eth1/31
Eth1/32 Eth1/32 Eth1/1
Eth1/1
IP ARP Table vPC
Eth1/1 Eth1/2
Total number of entries: 1
Address Age MAC Address Interface
IP=10.10.101.20/24
10.10.101.20 00:18:16 547f.ee1c.06fc Vlan888 Mac=547f.ee1c.06fc VLAn888

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Check Hardware Table On Module
N92160-1# slot 1 show hardware internal tah l3 10.10.101.20 table 1

DLeft location: 0x263c80

FP location : 0/0/0x18f2
HW Loc | Ip Entry | VRF | MPath | NumP | Base/L2ptr |CC|SR|DR|TD|DC|DE|LI|
-----------|-----------------------|--------|--------- -|----------|-----------------|----|----|----|-----|----|----|--|
0/0/0x18f2 | 10.10.101.20 | 1 | No |0 | 0x90091 | | | | |Y | | |
CC=Copy To CPU, SR=SA Sup Redirect,
DR=DA Sup Redirect, TD=Bypass TTL Dec,
DC=SA Direct Connect, DE=Route Default Entry,
AdjId | FP | BD | DMac | DstIdx | DstIsPtr | LI=Route Learn Info
-------------|--------------|--------|-------------------------|-----------|-------------|
0x90091 | 9/0/0x91 | 101 | 00:00:00:90:29:36 | 1538 | No |

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
ELAM on TOR with Cisco ASICs
module-1(TAH-elam)# debug platform internal tah elam asic 0
module-1(TAH-elam)# trigger init asic 0 slice 0 lu-a2d 1 in-select 6 out-select 0
module-1(TAH-elam-insel6)# set outer ipv4 src_ip 10.10.101.20 dst_ip 10.10.201.20
module-1(TAH-elam-insel6)# report Ver = 4, DSCP = 0, Don't Fragment = 0
DAVOS ELAM REPORT SUMMARY Proto = 1, TTL = 253, More Fragments = 0
========================= Hdr len = 20, Pkt len = 84, Checksum = 0xfe2c
Incoming Interface: Eth1/1 L4 Protocol :1
Src Idx : 0x602, Src BD : NA ICMP type : 0
Outgoing Interface Info: dmod 1, dpid 26 ICMP code : 0
Dst Idx : 0x3c, Dst BD : 4157 Drop Info:
----------
Packet Type: IPv4 Related show tech(s)
LUA:
Dst MAC address: 00:FE:C8:0E:27:63
LUB: N9xx#show tech-support forwarding l3 unicast
Src MAC address: 54:7F:EE:1C:06:FC
LUC: detail
1q Tag0 VLAN: 125, cos = 0x6 Nexus9500# show tech-support <service>
LUD:
Dst IPv4 address: 10.10.201.20 ACL_DROP
Src IPv4 address: 10.10.101.20 Final Drops: ACL_DROP

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Path of the Packet -Inband
CPU
• Traffic from all ingress Line Card
to Supervisor will hash to one
Netstack
Fabric module
• Traffic from Supervisor Card to
NIC-Eth2 NIC-Eth3 Egress Line cad will hash on one
FM. May not be same
Mod29 System Controller-SC1 • CoPP is operational on all LC.
However aggregate CoPP is on
FM

Fabric Module
Fabric Module

Fabric Module
Mod26
Mod21 Mod23

Eth6/1
Line Card OSPF Hello

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 Check for Drops/Errors-Line Card

N9500#show hardware internal interface asic counters mod 1

Important Counters/Drops
--------------- --------- --------- --------- --------- --------- ---------
Interface Name Drop Reasons for the Interface, See below output for detail if any
--------------- --------- --------- --------- --------- --------- --------------------
8 6 5 4
9-----------------------------------------------------------------------1110000000000
4 7 1
5-----------------------------------------------------------------------2109876543210
1
--------------- --------- --------- --------- --------- ---------------------
iEth1------------------------------------------------------------------------
X X
iEth2----------------------------------------------------------------------------------------------------
X X

41 : TAHOE Ingress DROP_VLAN_XLATE_MISS

51 : TAHOE Ingress DROP_SRC_VLAN_MBR
67 : TAHOE Ingress DROP_ACL_DROP
84 : TAHOE Ingress DROP_UC_DF_CHECK_FAILURE

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 Line Card
North Star ASIC
Check for Drops/Errors-Line Card Trident II
ASIC
Network Interfaces
N9K#show hardware internal interface ethernet 6/1 asic counters
Important Counters/Drops
--------------- --------- --------- --------- --------- --------- ---------
Interface Name Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
RxDrops TxDrops RxDrops TxDrops Drops Drops
--------------- --------- --------- --------- --------- --------- ---------
Ethernet6/1 870 0 100 0 0 0
--------------- --------- --------- --------- --------- --------- ---------
Forward Rx Drops = [ RDBGC0 RDBGC4 RDBGC6 RDBGC7 RDBGC8 ]
Forward Tx Drops = [ TDBGC1 TDBGC3 TDBGC5 (excludes expected Multicast drops)]
ErrorPkt Rx Drops= [ IUNHGI IUNKOPC RFCS RALN RFLR RERPKT RJBR RSCHCRC RUND RMTUE]
ErrorPkt Tx Drops= [ TJBR TFCS TRPKT RMTUE TUFL TPCE ]
QOS Rx Drops = [ RDISC DROP_PKT_ING DROP_PKT_IMTR DROP_PKT_YEL DROP_PKT_RED ]
QOS Tx Drops = [ MCQ_DROP_PKT(0) MCQ_DROP_PKT(1) MCQ_DROP_PKT(2)
Use slot <#> show hardware internal interface indiscard-stats instance <#> RDBGC0
N9K#bcm-shell mod 6 "listreg RALN"| grep Description
Description: Receive Alignment Error Frame Counter

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Instant Buffer Usage Stats-With Buffer Usage
N9K#show hardware internal buffer info pkt-stats mod 6
INSTANCE: 0

Output Shared Service Pool Buffer Utilization (in cells)

SP-0 SP-1 SP-2 SP-3
-------------------------------------------------------------------------
Total Instant Usage 4474 0 89 2939 • SP-3 Started filling
Remaining Instant Usage 25466 0 14255 3405 the Queue
------------------------------------------------------------------------
ASIC Port Q3 Q2 Q1 Q0 CPU SPAN
[13]
Only printed if there is congestion
UC(OOBFC)-> 0 0 0 0
UC-> 0 0 0 1249 332 0 • CPU buffer filling
MC-> 0 0 0 3247 1996 0 up

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 CoPP Drops on Line Card
Line Card
North Star ASIC
Trident II
ASIC
Network Interfaces

N9K# show policy-map interface control-plane mod 6 class copp-system-p-class-

critical | in ospf|trans|dropped
match access-group name copp-system-p-acl-ospf
transmitted 21898 packets;
dropped 0 packets;

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Identify FM -Check CoPP Drops

N9K# show hardware internal cpu-mac inband active-fm traffic-to-sup

Active FM Module for traffic to sup:
0x00000015 Fabric Module in Slot 21 carry all traffic to Sup

N9K# show policy-map interface control-plane mod 21 class copp-system-p-class-

critical | in ospf|trans|dropped
match access-group name copp-system-p-acl-ospf
match access-group name copp-system-p-acl-ospf6
transmitted 21898 packets;
dropped 0 packets;

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Check for Drops/Errors-Fabric Module
N9K# show system internal fabric connectivity mod 6 | grep 21 Identify HG Port on LC and FM
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink

6 0 HG10 3B 21 0 HG15
N9K# sh hardware internal fabric interface asic counters module 6 instance 0 asic-port 11

Important Counters/Drops Verify Drops/Error on HG port on LC

FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
RxDrops TxDrops RxDrops TxDrops Drops Drops
0 / 11 / HG10 0 0 0 0 0 0
N9K# sh hardware internal fabric interface asic counters mod 21 in 0 asic-port 16

RxDrops TxDrops RxDrops TxDrops Drops Drops

0 / 11 / HG15 0 0 0 0 0 0

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 Check for Drops/Errors-Fabric Module
Nexus9500# show system internal fabric connectivity mod 1 | grep 22 Identify HG Port on LC and FM
LC-Slot LC-Unit LC-iEthLink MUX FM-Slot FM-Unit FM-iEthLink
1 0 iEth03 - 22 0 iEth09
1 0 iEth05 - 22 1 iEth4
1 1 iEth11 - 22 0 iEth06

N9500#show hardware internal interface asic counters mod 22 EOR With

Important Counters/Drops LSE ASICs
--------------- --------- --------- --------- --------- --------- ---------
Interface Name Drop Reasons for the Interface, See below output for detail if any
--------------- --------- --------- --------- --------- --------- ----------------------
9-----------------------------------------------------------------------1110000000000
8
5-----------------------------------------------------------------------2109876543210
4
-----------------------------------------------------------------------------
iEth1------------------------------------------------------------------------
X
iEth2----------------------------------------------------------------------------------------------------
84 : TAHOE Ingress DROP_UC_DF_CHECK_FAILURE

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 Verify Drops Between FM and SC System Controller
MVDXN-SW

module-21# show mvdxn internal port-status MVDXN-SW

Switch type: Marvell 98DXN11 - 10 port switch Fabric Module in Slot 21 FABRIC CARD
Port Descr Enable Status ANeg Speed Mode InByte OutByte InPkts OutPkts
3 SC1EPCswitch Yes UP No 2 6 109548011 117051401 274144 587285
10 port switch on System
controller and Fabric
module-29# show mvdxn internal port-status module connect SC to FM

Switch type: Marvell 98DXN11 - 10 port switch System Controller in Slot 29

Port Descr Enable Status ANeg Speed Mode InByte OutByte InPkts OutPkts
7 FM1EPCswitch Yes UP No 2 6 746159513 60543666 620863 269592

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Drops/Errors On Supervisor
N9K#show hardware internal cpu-mac inband counters in eth|ps-inb|dro
Netstack
eth2 Link encap:Ethernet HWaddr 00:00:00:01:1b:01

RX packets:2922013 errors:0 dropped:0 overruns:2 frame:0 Pseudo Inband

TX packets:1652929 errors:0 dropped:0 overruns:0 carrier:0
NIC-Eth2 NIC-Eth3
eth3 Link encap:Ethernet HWaddr 00:00:00:01:1b:01
Supervisor Card
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

ps-inb Link encap:Ethernet HWaddr 00:00:00:01:1b:01

RX packets:54204 errors:0 dropped:3579 overruns:0 frame:0

TX packets:50626 errors:0 dropped:0 overruns:0 carrier:0

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Drops/Errors On Supervisor-Cont.
N9K#show hardware internal cpu-mac inband stats | in errors|rate|Queue

Queue Idx Packet Count Bytes Drops Csum Errors Allocation Failure

Queue 0 65429 580195964 2 0 0

Queue 7 65429 580195964 0 0 0

CRC errors ...................... 0

Alignment errors .............. 0

Symbol errors .................. 0

Related show tech(s)
Carrier extension errors ....0

Nexus9500# sh tech-support inband counters

Rx packet rate (current/peak) 812 / 1097 pps Nexus9500# show tech-support pktmgr
Nexus9500# show tech-support <service>
Tx packet rate (current/peak) 454 / 741 pps

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Virtual Port-Channel-vPC
• Allow a single device to use a port channel across
two upstream switches
• Eliminate STP blocked ports
• Dual-homed server operate in active-active mode
• HSRP-Both active and standby peers forward
packets-ARP response by Active
• Configuration steps Same as other Nexus Logical Topology with vPC
Products

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 Case:1 All vPC Leg UP
Scenario: Traffic of a Host in Vlan 10 connected to Switch-A hash to N9K1 to reach Host in Vlan 20
connected to Switch-B
N9k1 N9k2 vPC Peer Link =Eth1/1,4/1
Keep Alive
PC1-PeerLink
SVI10
SVI10
10.10.10.1/24 MCT-1/1, 4/1 10.10.10.2/24
SVI-Mac 78da.6e71.9a3f Eth6/20 Eth4/18
SVI-mac 003a.99fc.dd7f
Standby 10.10.10.3 Eth4/18 Eth6/20
Standby 10.10.10.3
HSRP-Mac 0000.0c07.ac0a
HSRP-Mac 0000.0c07.ac0a
SVI20
vPC10 vPC20 SVI20
SVI-mac 78da.6e71.9a3f
SVI-mac 003a.99fc.dd7f
10.10.20.1/24
Switch-A Switch-B 10.10.20.2/24
Standby 10.10.20.3
Standby 10.10.20.3
HSRP-Mac 0000.0c07.ac14
HOST-A Vlan-10 HOST-B Vlan-20 HSRP-Mac 0000.0c07.ac14

10.10.10.x/24 20.20.20.x/24

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
vPC-Router MAC Programming Check
• Both Active and Standby Peer responsible for L3 switching
• Virtual Mac address must be programmed in Hardware on Both peers
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan10 10 100 Active 10.10.10.2 local 10.10.10.3
N9K1# bcm-shell mod 4 "0:d chg my_station_tcam" | grep
VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,

Interface Grp Prio P State Active addr Standby addr Group addr
Vlan10 10 100 Standby 10.10.10.2 local 10.10.10.3
N9K2# bcm-shell mod 4 "0:d chg my_station_tcam" | grep
VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
vPC Peer Gateway Programming Check
• Are N9K’s Configured with Peer-Gateway
N9K1-SJ# show mac address-table vlan 10 | in G
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
G 10 0000.0c07.ac0a static - F F vPC Peer-Link(R)
G 10 003a.99fc.dd7f static - F F sup-eth1(R) N9K2 SVI MAC
G 10 78da.6e71.9a3f static - F F vPC Peer-Link®

N9K# bcm-shell mod 4 "0:d chg my_station_tcam" | egrep 0x003a99fcdd7f

MY_STATION_TCAM.ipipe0[0]:
<VALID=1,MAC_ADDR_MASK=0xffffffffffff,MAC_ADDR=0x003a99fcdd7f,KEY=0x00000000003
a99fcdd7f,IPV6_TERMINATION_ALLOWED=1,IPV4_TERMINATION_ALLOWED=1,DATA=0x38,ARP_R
ARP_TERMINATION_ALLOWED=1>

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
vPC Check For Traffic Ingressing Peer Link
Egress Block Mask
• vPC Check-Traffic from Peer Link should Not L2/L3 Switch with local and remote
Legs up
N9K1# show vpc brief | grep Po
id Port Status Active vlans N9k1 N9k2
1 Po1 up 10-20 Keep Alive

id Port Status Consistency Reason Activevlans PC1-PeerLink

10 Po10 up success success 10-20 MCT-1/1, 4/1

20 Po20 up success success 10-20 Eth6/20 Eth4/18
Eth4/18 Eth6/20
N9K2# show vpc brief | grep Po
id Port Status Active vlans vPC10 vPC20
1 Po1 up 10-20
id Port Status Consistency Reason Activevlans Switch-A Switch-B
10 Po10 up success success 10-20
20 Po20 up success success 10-20

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
vPC Check for Traffic Ingressing Peer Link (Cont’d)

N9K1#show port-ch summary | in Po N9k1 Keep Alive N9k2

Group Port- Type Protocol Member Ports PC1-PeerLink
1 Po1(SU) Eth LACP Eth1/1(P)Eth4/1(P) MCT-1/1, 4/1
10 Po10(SU)Eth LACP Eth4/18(P) Eth6/20 Eth4/18
20 Po20(SU)Eth LACP Eth6/20(P) Eth4/18 Eth6/20

vPC10 vPC20
N9K1# show system internal vpcm info mask
module 6 Switch-A Switch-B
Masked ports for Module 6, Unit 0:
[Src Port None]: Eth6/20 Traffic Ingressing on Eth1/1 and
[Src Port Eth1/1]: Eth6/20
[Src Port Eth4/1]: Eth6/20 Eth4/1 will not exit Eth 6/20
Masked ports for Module 6, Unit 1:

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
 ACL redirect logic for routed packets-vPC Leg Down
• Redirect ACL installed to redirect routed packets for the
vPC for which local interface goes down N9k1 N9k2
Keep Alive
• Mac address learned from vPC points virtual port PC1-PeerLink

MCT-1/1, 4/1
N9K1# show hardware access-list tcam region | grep vpc Eth6/20 Eth4/18
VPC Convergence [vpc-convergence] size = 512 Eth4/18 Eth6/20

N9K1# sh mac address-table address30f7.0d9b.d401

Link Down
vPC10 vPC20
VLAN MAC Address Type age Secure NTFY Ports

20 30f7.0d9b.d401 dynamic 0 F F vPC Peer-Link

Switch-A Switch-B

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
ACL redirect logic for routed packets-Verify TrunkID

N9508a-SJ# show system internal access-list vpc-convergence mod 6

------------------------------------------------------------
VPC Convergence Entries
------------------------------------------------------------
Instance: 0
========== Trunk-id of “3” Down vPC
Ingress:
---------- Trunk-id of vPC Peerlink
Entry-ID DstTrunk-GID RedirectTrunk-GID Packet-Count
------------------------------------------------------------------------
1539 3 1 6082015

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
ACL redirect logic for routed packets-vPC Leg Down
Mod1#debug hardware internal lac dump asic 0 slice 0 table tah_lac_luc_localmultipathtable 1538 1 field-per-line | in is|bounce

is_vpc=0x00000001 Entry# is 1536+dpid

is_vpc=0x00000000
is_vpc=0x00000000 vpc_bounce=0x00000000
vpc_bounce=0x00000001
vpc_bounce=0x00000001 N9k1 N9k2
Keep Alive
ecc=0x00000000
PC1-PeerLink
With All Interface of vPC UP
No Redirect Traffic to MCT Drop Traffic coming from MCT
MCT-1/1, 4/1
is_vpc=0x00000001 is_vpc=0x00000001 Eth6/20 Eth4/18
vpc_bounce=0x00000000 vpc_bounce=0x00000000
Eth4/18 Eth6/20
show system internal ethpm info interface po101 | in STATIC
IF_STATIC_INFO: port_name=port-channel101,nxos_port=19635,dmod=0,dpid=2, Link Down
vPC101 vPC102

show tech-support vPC Switch-A Switch-B

show tech-support cfs
show tech-support port-channel
BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
 Agenda
• Introduction
• Architecture-Brief
• Troubleshooting Toolkit
• Nexus 9000 Troubleshooting
• Common Link Layer Issues-L1
• L2/L3 Packet Forwarding
• Path of the Packet to CPU
• vPC Troubleshooting

• Nexus9000 Specific Limitation

and Giddies
Email from Nexus9000 To Cisco SR
• Commands output directly sent to email address
• Information from Nexus9000 Can be directly attached to Service Request.
• Information is sent as body to email- not as attachment
N9K(config)# email
N9K(config-email)# smtp
N9K(config-email)# smtp-host 173.37.37.37
N9K(config-email)# from [email protected]
N9K(config-email)# smtp-port 25
show run | email subject <SR-number> [email protected]

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Copying a File faster to the bootflash through kstack
• TFTP/FTP/ SFTP/SCP for copying a file to the bootflash on Nexus 9000 and its slow in nxos- uses
netstack

• Using bash to copy the files it is way faster-bash copy uses kstack
• Nexus9500# conf ter
Enter configuration commands, one per line. End with CNTL/Z.
Nexus9500 (config)# feature bash
Nexus9500 (config)# end
Nexus9500# run bash
bash-4.2$ sudo su -
• root@VTEP_TOP2#ip net exec management scp [email protected]:/auto/user/images
/final/nxos.7.0.3.I4.6.bin /bootflash/
This is your AD password: *******
File got copied in 33 Sec
nxos.7.0.3.I4.6.bin 100% 650MB 20.2MB/s 00:33

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Bash Support !!!!
• Goes beyond what standard CLI can provide
• Customers demand more capabilities/freedom Creativity

• Feature: bash-shell
• User Role: dev-ops or network-admin or vdc-admin*

• Strongly recommended: Some experience with shell/Linux-Use with extreme care

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Broadcom ASIC shell access on the Nexus 9000 !!!
• The Nexus 9000 is based largely on the Broadcom Trident II ASIC-Known as T2

• The modular unit Fabric Modules (FM) and Line Cards (LC) each contain multiple
instances of the T2 ASIC, as well as the TOR (top of rack) units

• Access is provided to each and every instance of the T2 ASIC

• No additional license is required to access the bcm-shell

• Permitted by default role network-admin

• Role based access control (RBAC) can be used to limit user access

• Accounting log available for BCM activity

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Python !!!!
• Python is - Established, Modern and Powerful, Clean, lots of libraries, liberal
license
• Perl is available in gdb images only – not available in final images
• Tcl is there but no one uses it in NX-OS
• The license that Python has (GPL-Like with very few restrictions on modification,
distribution and commercial use) make it very attractive to embed and distribute

• On the box applications that can currently use Python scripts

• Embedded Event Manager
• Power On Auto Provisioning (POAP)
• Create your own scripts that are like “Super commands”
• Create your own command modifiers – the things that act on commands applied with a
pipe “|”

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Python Script Example

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Important Limitations
• For every Feature please review Guidelines and Limitations
• Cisco Nexus 9000 Series NX-OS Verified Scalability Guide
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/scalability/guide_703I52/b_Cisco_Nexus_9000_Series_NX-
OS_Verified_Scalability_Guide_703I52.html?referring_site=RE&pos=1&page=http://www.cisco.co
m/c/en/us/td/docs/switches/da
• EPLD Upgrade are recommended but are not mandatory
• User Configured MAC address for SVI- Packets will not be flooded if Layer 2 Adjacency is missing
• ASIC Memory-NS test is applicable only for the N9K-X9564PX and N9K-X9564TX line cards.
• Priority flow control (PFC) is supported on Cisco Nexus 9500 Series switches with the N9K-
X9636PQ line card.
• Cisco Nexus 9500 Series Switch can run in 8-queue mode only if all of its line cards are capable of
running 8-queue mode.

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
 Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions

BRKDCN-3101 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Thank you
Miscellaneous
Glossary of Terms
NFE Network Forwarding Engine (Trident2 ASIC)
NFE2 Network Forwarding Engine 2 (Tomahawk ASIC)
DMOD Destination Module (ASIC Instance)
DPORT Destination Port (Port within ASIC instance)
UFT Unified Forwarding Table
LC Line card Module
FM Fabric Module
LPM Longest Prefix Match (e.g IPv4 : /31 to /0, IPv6 /127 to /0)
ALPM Algorithmic Longest Prefix Match
ECMP Equal Cost Multipath

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
bcm-shell command reference
Command Usage
l3 l3table show Display IPv4 Host Entries

l3 defip show Display IPv4 LPM Entries

l3 ip6host show Display IPv6 Host Entries

l3 ip6route show Display IPv6 LPM Entries

l3 multipath show Display ECMP object and Nexthop List

l3 egress show Display Egress object with rewrite information

show c rpkt Display Received packets in Hardware

show c tpkt Display Transmitted packets in Hardware

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
Elam Ingress & Egress Direction-EOR

• Traffic entering from Fabric Module in to NS of Line Card is Fabric 3

Egress Pipeline
Fabric 1
N N
Egress FE FE
Ingress
Ex. trigger init egress in-select 3 out-select 5
set outer ipv4 dst_ip 13.13.13.10 Line Card

North Star ASIC

• Traffic Entering NS and exiting towards Fabric Module is 12 x 40G
Hi-Gig2
Ingress Pipeline Trident II
ASIC
12 x 40G
Ex. trigger init ingress in-select 3 out-select 5 Ethernet
Network Interfaces
set outer ipv4 src_ip 13.13.13.10
Front Panel 48x 1GE/10GE Ports

13.13.13.10

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
 Important ELAM Fields

GBL_C++ [INFO] hg2_srcmod: 0E Information

GBL_C++ [INFO] hg2_srcpid: 0D
is in Hex N9K# show interface hardware-mappings
Convert to -------------------------------------------
GBL_C++ [INFO] hg2_dstmod: 11 Dec. ----------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort
GBL_C++ [INFO] hg2_dstpid: 0A
------------------------------------------
GBL_C++ [INFO] ip_da: 000000000000D0D0D0A Eth5/2 1a280000 14 0 13 255 0 -1
GBL_C++ [INFO] ip_sa: 000000000000D0D0D01 Eth6/52 1a286600 17 1 10 255 51 -1

GBL_C++: [MSG] - sideband is complete Sideband is the result where

GBL_C++: [INFO] ovector: 000FFF packet will be sprayed.
Should never be “0”

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
N9K-X9636PQ

FM1 FM2 FM3 FM4 FM5 FM6

HG Ports HG Ports HG Ports

T2 T2 T2
Instance 0 Instance 1 Instance 2

QSPF Ports QSPF Ports QSPF Ports

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
N9K-X9464PX

FM2 FM3 FM4 FM6

MUX1-2 MUX3-4

HG Ports HG Ports
T2

10G SFP+ Ports 40G QSFP

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
N9K-X9464TX

FM2 FM3 FM4 FM6

MUX1-2 MUX3-4

HG Ports HG Ports
T2

100/1000/10000 T Ports 40G QSFP

10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G FP FP FP FP
PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
49 50 51 52
FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
N9K-X9432PQ

FM2 FM3 FM4 FM6

HG Ports HG Ports

T2 T2
Instance 0 Instance 2

QSPF Ports QSPF Ports

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
 N9K-X9564PQ
FM6 FM5 FM4 FM3 FM2 FM1

HG MUX1 HG MUX4 HG MUX2 HG MUX5 HG MUX3 HG MUX6

0123 4567 8 9 10 11 0123 4567 8 9 10 11

MN Port MN Port
Northstar 1 Northstar 2
MF Port MF Port
0-2 3-5 6-8 9-11 0-2 9-11

7-5 2-0 31-29 26-24 7-5 26-24

Warpcore
T2 T2

40G QSFP

10G SFP+ Ports FP FP FP FP

49 50 51 52
FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
N9K-X9564TX
FM6 FM5 FM4 FM3 FM2 FM1

HG MUX1 HG MUX4 HG MUX2 HG MUX5 HG MUX3 HG MUX6

0123 4567 8 9 10 11 0123 4567 8 9 10 11

MN Port MN Port
Northstar 1 Northstar 2
MF Port MF Port
0-2 3-5 6-8 9-11 0-2 9-11

7-5 2-0 31-29 26-24 7-5 26-24

T2 T2

40G QSFP
100/1000/10000 T Ports
10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G
PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
FP FP FP FP
49 50 51 52
FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 109