Multicast Troubleshooting

Denise “Fish” Fishburne

Customer Proof of Concept Team Lead

CCIE #2639, CCDE 2009:0014

BRKIPM-2264
Agenda

• Quick Review

• Multicast Troubleshooting Methodology

• Troubleshooting ASM

• Troubleshooting SSM

3
Housekeeping Items
• Cell Phones

• “Intermediate” Class
• Assumes Multicast Operational Experience

• Questions

4
Quick Review

5
Quick Review Agenda

• Frequently Used Terms

• Tree Growth
• Sending a (*,G) PIM Join
• Registering a Source
• Designated Routers

6
Frequently Used Terms
ASM - Any Source Multicast

SSM - Source Specific Multicast

MDT - Multicast Distribution Tree

RP - Rendezvous Point

FHR - First Hop Router

LHR - Last Hop Router

IR - Intermediate Router

DR - Designated Router

IGMP- Internet Group Management Protocol

7
Frequently Used Terms
• Any Source Multicast (ASM)
• Original (Classic) PIM-SM
• Supports both Shared and Source Trees

• Source Specific Multicast (SSM)

• a.k.a. Single Source Multicast
• Supports only Source Trees

8
Frequently Used Terms
(*,G) = *,G
= Shared Tree
= RP tree

*,G MDT

RP

LHR

9
Frequently Used Terms
(S,G) = S,G
= Shortest Path Tree
= Source tree

S,G MDT

FHR
LHR

10
Frequently Used Terms
(*,G) = *,G
= Shared Tree
= RP tree

(S,G) = S,G
= Source Tree
= Shortest Path Tree

Q: What is the Root of the (*,G) Tree?

A: RP
Q: What is the Root of the (S,G) Tree?
A: Source
11
Frequently Used Terms
IIF = Incoming Interface

Interface towards the root of the tree

OIL = Outgoing interface List

Interfaces that have received PIM Joins or IGMP membership requests

Q: What does the IIF of the (*,G) Tree point towards?

A: RP
Q: What does the IIF of the (S,G) Tree point towards
A: Source
12
Tree Growth
Nature’s Way

In Nature, trees grow

• UP and
• OUT

from the ROOT

13
Tree Growth
Multicast Distribution Tree (MDT)

With Multicast, trees grow

• TOWARD

the ROOT

14
 Sending a (*,G) PIM Join

10.4.5.0
.5 .4
PIM Join
Router A Router B RP

Question:
When Router A sends a PIM (*,G) Join out, what will be the Destination IP?
IP address of the RP? 224.0.0.13? IP of Router B
(All PIM Routers) (10.4.5.4)

**Assume Router A knows the RP, has the RP in its RIB via an IGP with B and has a PIM
neighbor with Router B

15
 Sending a (*,G) PIM Join
The PIM Header
PIM header common to all PIM messages:

Some PIM Types are:

Message Type Destination
0 = Hello Multicast to ALL-PIM-ROUTERS

1 = Register Unicast to RP
2 = Register-Stop Unicast to source of Register packet
3 = Join/Prune Multicast to ALL-PIM-ROUTERS
4 = Bootstrap Multicast to ALL-PIM-ROUTERS
5 = Assert Multicast to ALL-PIM-ROUTERS
http://www.ietf.org/rfc/rfc4601.txt
16
 Sending a (*,G) PIM Join

10.4.5.0
.5 .4
Router A Router B RP
DIP 224.0.0.13 PIM Join

Q: When Router A sends a PIM (*,G) Join out, what will be the Destination IP?

IP of the RP?
224.0.0.13? IP of Router B
(All PIM Routers) (10.4.5.4)

17
 Sending a (*,G) PIM Join

A B
.5 10.4.5.0 .4

RP
DIP 224.0.0.13 PIM Join
.1

C
Question:
If the destination IP address of the Join is 224.0.0.13 (all PIM Routers), then
how do we keep Router B and Router C from BOTH acting on that join?

18
 Sending a (*,G) Join
A B
.5 10.4.5.0 .4

DIP 224.0.0.13 PIM Join

RP
.1

C
Answer:
The address of the upstream neighbor (B or C) is the target of the message

http://www.ietf.org/rfc/rfc4601.txt
19
Sending a (*,G) PIM Join
A B
.5 10.4.5.0 .4

PIM Join

.1

Destination IP C
224.0.0.13

IP address of B
10.4.5.4

RP
2.2.2.2
20
 ***When a shared MDT is already
Registering a Source*** active for that group on the RP
FHR RP = 2.2.2.2

gig0
Mcast packet is encapsulated into unicast PIM
Mcast Data packet:
Dest IP=RP PIM Mcast data Destination IP is the RP.
PIM header type is Register
gig0 added to the RP sends PIM (S,G) Join Back Towards Source
OIL for (S,G) if there is an active shared tree for that G
Mcast packets now get sent out twice
Mcast data 1.With Register
2.Out (S,G)
Mcast data Once the RP sees the packets come in
on the (S,G),
…. it sends a unicast Register Stop
Register Stop to the FHR
Mcast data Mcast data Mcast packets now only out the (S,G) tree
21
Designated Routers (DRs)
Router A
.3

Vlan
FHR Q: Which router should register the source
with the RP?
.2
A: The FHR that is the DR
Router B

22
Designated Routers (DRs)
.3

Vlan
Q: Which router should send the PIM (*,G)
Join?
.2
A: The LHR that is the DR
LHR

23
Multicast Troubleshooting
Methodology

24
Multicast Troubleshooting Methodology Agenda

• The Plumbing

• Troubleshooting Toolbox

25
Building the Plumbing
Unicast

EIGRP EIGRP EIGRP

R1 R1 Subnet
R1 R1
10.1.1.0/24
advertise advertise advertise
Ping
10.1.1.0/24 10.1.1.0/24 10.1.1.0/24
10.1.1.1 10.1.1.1

Point A Point B

26
 Building the Plumbing ***In this example, LHR is configured
to not cut over to the S,G tree.
Multicast***

PIM PIM PIM PIM

R1 R1 R1 R1
FHR RP PIM (*,G) PIM (*,G) LHR
Source PIM Unicast Register Join Join Join
(Tx)

PIM (S,G) PIM (S,G)

Join Join

27
Sherlock Holmes

28
Troubleshooting the ASM Plumbing LHR

FHR RP

MDTs Built Backwards Towards Root

Traffic

Troubleshooting

29
Troubleshooting the SSM Plumbing LHR

FHR

MDTs Built Backwards Towards Root

Traffic

Troubleshooting

30
Troubleshooting Toolbox
“Go To” Commands
show ip mroute
show ip mroute count

show ip pim neighbor

show ip pim interface

show ip igmp group

show ip pim rp [group]

show ip pim rp mapping [group]
show ip rpf [address]
31
Troubleshooting Toolbox
Facts to Remember

• Multicast Trees get built backwards towards the root

• Multicast traffic in ASM and SSM are triggered via a “pull”

• High on the “food chain” dependent on routing working and PIM working

32
Troubleshooting Toolbox
Questions to Remember

• Who is the root?

• Where is the root?

• What is the PIM RPF neighbor towards the root?

33
Troubleshooting Toolbox
Checklist

 IGMP Membership Report Received by LHR

 “WHO:” is the root?
 “WHERE” is the root?
 “WHAT” is the PIM RPF neighbor towards the root?

34
ASM Troubleshooting

35
ASM Troubleshooting
R3
R3

R1
R1 R2
R2 R5
R5

FHR RP LHR
R4

(*,G) from LHR towards the RP 1

FHR registers source with RP

2
RP sends PIM (S,G) Join Towards Source
3
LHR Cutting over to SPT (S,G)
36
ASM Troubleshooting
Shared Tree
R3
R3

R1
R1 R2
R2 R5
R5

FHR RP LHR
R4

(*,G) from LHR towards the RP 1

FHR registers source with RP
2
3
RP sends PIM (S,G) Join Towards Source
LHR Cutting over to SPT (S,G)
37
ASM Troubleshooting:
Shared Tree
R3
RP LHR

R2 R5
1a
LHR receives
1c R4
IGMP Membership Report
R4 (an IR) sends a PIM (*,G) 1b
To R2 (RP)
LHR (R5) sends a PIM (*,G)
To R4

(*,G) from LHR towards the RP 1

38
ASM Troubleshooting:
Shared Tree
R3
RP LHR

R2 R5
1a
R4 LHR receives
IGMP Membership Report
 IGMP Membership Report Received by LHR
 “WHO:” LHR knows who the RP is for this group
 “WHERE:” LHR knows where (RIB) the RP is
 “WHAT:” LHR has a PIM Neighbor to send the PIM (*,G) Join to

39
ASM Troubleshooting:
Shared Tree
1a LHR Receives IGMP Membership Report LHR

R5

 IGMP Membership Report Received by LHR

LHR#show ip mroute Is completely empty

LHR#show ip igmp group Is completely empty
LHR#show ip igmp interface Is completely empty

40
ASM Troubleshooting:
Shared Tree
1a LHR Receives IGMP Membership Report LHR

R5
 IGMP Membership Report Received by LHR
• PIM not enable on interface of Receiver
• Layer 1 issues
• Layer 2 issues
• Access-List
• IGMP membership report not being sent

LHR#show run interface gig1/1 interface GigabitEthernet1/1

no switchport
ip address 100.1.1.1 255.255.255.0
ip pim sparse-mode

41
 ASM Troubleshooting:
Shared Tree
1a LHR Receives IGMP Membership Report LHR

R5

 IGMP Membership Report Received by LHR

LHR#show run interface gig1/1 interface GigabitEthernet1/1

no switchport
ip address 100.1.1.1 255.255.255.0
LHR#show ip igmp groups
ip pim sparse-mode
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter
239.1.1.1 GigabitEthernet1/1 00:18:35 00:02:31 100.1.1.100
42
 ASM Troubleshooting:
Shared Tree
1b LHR Sends PIM (*,G) Join R3
LHR

R2 R5

RP
R4 1b
LHR (R5) sends a PIM (*,G)
To R4
 “WHO:” LHR knows who the RP is for this group
 “WHERE:” LHR knows where (RIB) the RP is
 “WHAT:” LHR has a PIM Neighbor to send the PIM (*,G) Join to

43
ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join

R2 R5
RP
Who? RP LHR
R4
Where?
RPF Nbr?

LHR#show ip mroute (*, 239.1.1.1), 00:01:23/00:02:48, RP 0.0.0.0, flags: SJC

Incoming interface: Null, RPF nbr 0.0.0.0 ?
Outgoing interface list:
GigabitEthernet1/1, Forward/Sparse, 00:01:23/00:02:48, H
LHR#show ip pim rp 239.1.1.1 Group: 239.1.1.1, RP: 0.0.0.0
?
44
ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join

Note:
R2 R5
RP information can be shared 3 ways
• Static RP LHR
R4
• Auto-rp
• BSR

LHR(config)#ip pim rp-address 2.2.2.2 ?

<1-99> Access-list reference for group
<1300-1999> Access-list reference for group (expanded range)
WORD IP Named Standard Access list
override Overrides dynamically learnt RP mappings

45
ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join

R2 R5

RP
✓
Who?
RP
R4 LHR
Where?
RPF Nbr?

LHR#sh ip pim rp 239.1.1.1

Group: 239.1.1.1, RP: 2.2.2.2, uptime 00:00:39, expires never

LHR#sh ip pim rp mapping 239.1.1.1

PIM Group-to-RP Mappings
Group(s): 224.0.0.0/4, Static-Override
RP: 2.2.2.2 (?) 46
ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join

R2 R5
RP
✓
Who?
RP LHR
Where? R4
RPF Nbr?

LHR#show ip mroute ✓
(*, 239.1.1.1), 00:08:18/00:02:46, RP 2.2.2.2, flags: SJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
GigabitEthernet1/1, Forward/Sparse, 00:08:18/00:02:46, H
LHR#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2) failed, no route exists

LHR#show ip route 2.2.2.2 % Network not in table

47
ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join

R2 R5
RP
✓
Who?
✓
Where? RP
R4 LHR
RPF Nbr?

LHR#show ip route 2.2.2.2 2.0.0.0/32 is subnetted, 1 subnets

O 2.2.2.2 [110/42] via 10.4.5.4, Gig 1/5
[110/42] via 10.3.5.3, Gig 1/4

48
ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join

RP R2 R5
✓
Who?
✓
Where? RP
R4 LHR
RPF Nbr?

LHR#show ip mroute
(*, 239.1.1.1), 00:08:18/00:02:46, RP 2.2.2.2, flags: SJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
GigabitEthernet1/1, Forward/Sparse, 00:08:18/00:02:46, H
49
 ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join

R2 R5
RP
✓
Who?
✓
Where? RP LHR
RPF Nbr? R4

LHR#show ip rpf 2.2.2.2

RPF information for ? (2.2.2.2) failed, no route exists
LHR#show ip pim neighbor No PIM neighbors listed

50
 ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join

R2 R5

RP LHR
R4

LHR#show ip pim neighbor

PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
P - Proxy Capable, S - State Refresh Capable
Neighbor Interface Uptime/Expires
10.3.5.3 GigabitEthernet1/4 00:17:57/00:01:21
10.4.5.4 GigabitEthernet1/5 00:17:56/00:01:31
51
 ASM Troubleshooting:
Shared Tree
R3
1b LHR Sends PIM (*,G) Join
Question:
Why was 10.4.5.4 chosen as the RPF neighbor? R2 R5
Answer:
If ECMP are available, the RPF for the multicast traffic RP LHR
R4
will be based on highest IP address for the RPF
neighbor

LHR#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2)

RPF interface: GigabitEthernet1/5
RPF neighbor: ? (10.4.5.4)
RPF route/mask: 2.2.2.2/32
RPF type: unicast (ospf 1)

52
 ASM Troubleshooting:
Shared Tree
1b LHR Sends PIM (*,G) Join R3

Note:
LHR now has all the information it needs to
R2 R5
send a PIM (*,G) join to its RPF neighbor.

RP RP
✓
Who? R4 LHR
✓
Where? 1b
✓ Nbr?
RPF LHR (R5) sends a PIM
(*,G) To R4

LHR#show ip mroute RP flags

*,G (*, 239.1.1.1), 00:36:39/00:02:56, RP 2.2.2.2, flags: SJC
IIF Incoming interface: Gig 1/5, RPF nbr 10.4.5.4
OIL Outgoing interface list: RPF nbr
Gig 1/1, Forward/Sparse-Dense, 00:08:08/00:02:56, H
53
 ASM Troubleshooting:
Shared Tree
1c IR Sends PIM (*,G) Join
R3

R2 R5

RP LHR
R4
R4 (an IR) sends a PIM (*,G) R4
To R2 (RP) 1c

 “WHO:” IR knows who the RP is for this group

 “WHERE:” IR knows where (RIB) the RP is
 “WHAT:” IR has a PIM Neighbor to send the PIM (*,G) Join to
54
 ASM Troubleshooting:
Shared Tree
1c IR Sends PIM (*,G) Join
R3
Note:
Telnetting backwards towards the root is
a good way to troubleshoot. R2 R5

RP LHR
LHR#show ip mroute 239.1.1.1 R4
R4
(*, 239.1.1.1), 00:36:39/00:02:56, RP 2.2.2.2, flags: SJC
Incoming interface: GigabitEthernet1/5, RPF nbr 10.4.5.4
Outgoing interface list:
Gig 1/1, Forward/Sparse-Dense, 00:08:08/00:02:56, H
LHR#telnet 10.4.5.4 RP
Trying 10.4.5.4 ... Open
R4#show ip mroute 239.1.1.1 Who?
✓
(*, 239.1.1.1), 00:48:49/00:02:33, RP 2.2.2.2, flags: S Where?
✓
Incoming interface: Gig 3/3, RPF nbr 10.2.4.2, RPF-MFD
Outgoing interface list: RPF
✓ Nbr?
Gig 3/5, Forward/Sparse, 00:47:16/00:02:33, H 55
ASM Troubleshooting:
Shared Tree
1c IR Sends PIM (*,G) Join
R3

Note:
R4 had all the information it needed to
send a PIM (*,G) join to its RPF
R2 R5
neighbor.
RP LHR
R4
R4#telnet 10.2.4.2 R4 (an IR) sent a PIM (*,G)R4
Trying 10.2.4.2... Open To R2 (RP) 1c
R2#show mrib route 239.1.1.1
(*,239.1.1.1) RPF nbr: 2.2.2.2 Flags: C
Up: 00:52:12
Incoming Interface List
Decapstunnel0 Flags: A, Up: 00:52:12
Outgoing Interface List
GigabitEthernet0/0/0/3 Flags: F NS, Up: 00:52:12
56
ASM Troubleshooting:
Shared Tree
R3
RP LHR

R2 R5
1a
LHR receives
1c R4
IGMP Membership Report
R4 (an IR) sends a PIM (*,G) 1b
To R2 (RP)
LHR (R5) sends a PIM (*,G)
(*,G) Between RP and LHR To R4

✓ (*,G) from LHR towards the RP 1

57
ASM Troubleshooting
R3
R3

R1
R1 R2
R2 R5
R5

FHR RP LHR
R4

(*,G) from LHR towards the RP 1

FHR registers source with RP
2
3
RP sends PIM (S,G) Join Towards Source
LHR Cutting over to SPT (S,G)
58
ASM Troubleshooting:
Source Tree
R1
R1 R2
R2

FHR RP
2a
FHR receives 2b
Multicast Data FHR registers source with RP
2c
RP builds S,G Back Towards Source

FHR registers source with RP

2
RP builds S,G Back Towards Source
59

***When a shared MDT is already active for that group on the RP

ASM Troubleshooting:
Source Tree
R1
R1 R2
R2

FHR RP
2a
FHR receives 2b
Multicast Data FHR registers source with RP
2c
RP builds S,G Back Towards Source

FHR registers source with RP

2
RP builds S,G Back Towards Source
60

***When a shared MDT is already active for that group on the RP

ASM Troubleshooting:
Source Tree
2a FHR Receives Multicast Data

R1
R1 R2
R2

FHR RP
2a
FHR receives
Multicast Data

2a
FHR receives Multicast Data

61
 ASM Troubleshooting:
Source Tree
2a FHR Receives Multicast Data R1
R1 R2
R2

 FHR Receives Multicast Data S: 200.1.1.100 RP

FHR
R1 R2
G: 239.1.1.1
• PIM not enable on interface of Receiver
• Layer 1 issues
2a
• Layer 2 issues
• Access-List FHR receives
• IGMP membership report not being sent Multicast Data
R1#show ip mroute Empty

R1#show run interface gig3/2 interface GigabitEthernet3/2

no switchport
ip address 200.1.1.1 255.255.255.0
ip pim sparse-mode
62
ASM Troubleshooting:
Source Tree
2a FHR Receives Multicast Data R1
R1 R2
R2

Note: S: 200.1.1.100 RP
We see that a (*,G) for that group has been G: 239.1.1.1 FHR
built.

R1#show ip mroute (*, 239.1.1.1), 00:02:17/00:00:42, RP 0.0.0.0, flags: SP

Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null

63
ASM Troubleshooting:
Source Tree
R1
R1 R2
R2

FHR RP
2a
FHR receives 2b
Multicast Data FHR registers source with RP
2c
RP builds S,G Back Towards Source

FHR registers source with RP

2
RP builds S,G Back Towards Source
64

***When a shared MDT is already active for that group on the RP

ASM Troubleshooting:
Source Tree
2b FHR Registers Source with RP

R1
R1 R2
R2

FHR RP

2b
FHR registers source with RP

2b
FHR registers source with RP

65
ASM Troubleshooting:
Source Tree
2b FHR Registers Source with RP R2
R2
R1
R1

Note: S: 200.1.1.100 RP
We see that a (*,G) for that group has been built. G: 239.1.1.1 FHR

Question:
Why is there no (S,G)?

R1#show ip mroute (*, 239.1.1.1), 00:02:17/00:00:42, RP 0.0.0.0, flags: SP

Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null

66
ASM Troubleshooting:
Source Tree
2b FHR Registers Source with RP R2
R2
R1
R1

Note: S: 200.1.1.100 RP
We see that a (*,G) for that group has been built. G: 239.1.1.1 FHR

Question:
Why is there no (S,G)?

R1#show ip mroute 239.1.1.1 count

IP Multicast Statistics
2 routes using 1230 bytes of memory
2 groups, 0.00 average sources per group
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second
Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)

Group: 239.1.1.1, Source count: 0, Packets forwarded: 0, Packets received: 6027

RP-tree: Forwarding: 0/0/0/0, Other: 6027/6026/1
RPF failed
67
ASM Troubleshooting:
Source Tree
2b FHR Registers Source with RP R2
R2
R1
R1

S: 200.1.1.100 RP
Problem:
G: 239.1.1.1 FHR
FHR does not know who the RP is for
this group.
RP
Who?
Where?
RPF Nbr?

R1#show ip mroute (*, 239.1.1.1), 00:02:17/00:00:42, RP 0.0.0.0, flags: SP

Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null
R1#sh ip rpf 2.2.2.2
RPF information for ? (2.2.2.2) failed, no route exists

68
ASM Troubleshooting:
Source Tree
2b FHR Registers Source with RP
R1
R1 R2
R2
Note:
RP information can be shared 3 ways S: 200.1.1.100 RP
• Static G: 239.1.1.1 FHR
• Auto-rp
• BSR
RP
✓
Who?
Where?
RPF Nbr?
R1#sh ip pim rp mapping 239.1.1.1
“ip pim rp-address 2.2.2.2 override” PIM Group-to-RP Mappings
Group(s): 224.0.0.0/4, Static-Override
RP: 2.2.2.2 (?)
✓
R1#show ip mroute (*, 239.1.1.1), 00:02:17/00:00:42, RP 2.2.2.2, flags: SP
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null

69
ASM Troubleshooting:
Source Tree
2b FHR Registers Source with RP R1
R1 R2
R2

S: 200.1.1.100 RP
Problem:
G: 239.1.1.1 FHR
FHR doesn’t know who the RPF
neighbor is for the shared tree for this
group. RP
✓
Who?
✓
Where?
RPF Nbr?
R1#show ip route 2.2.2.2 2.0.0.0/32 is subnetted, 1 subnets
O 2.2.2.2 [110/2] via 10.1.2.2, 01:11:17, Gig3/1

R1#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2) failed, no route exists

R1#show ip pim neighbor No PIM neighbors

70
ASM Troubleshooting:
Source Tree
2b FHR Registers Source with RP R1
R1 R2
R2

S: 200.1.1.100 RP
Note:
G: 239.1.1.1 FHR
“ip pim sparse-mode” added to Gig3/1 on
FHR (R1)

R1#show ip mroute
(*, 239.1.1.1), 00:02:45/stopped, RP 2.2.2.2, flags: SPF
Incoming interface: Gig3/1, RPF nbr 10.1.2.2, RPF-MFD
Outgoing interface list: Null

(200.1.1.100, 239.1.1.1), 00:01:34/00:02:59, flags: PFT

✓
Incoming interface: Gig3/21, RPF nbr 0.0.0.0, Registering (data-header), Partial-SC
Outgoing interface list: Null

71
ASM Troubleshooting:
Source Tree
R1
R1 R2
R2

FHR RP
2a
FHR receives 2b
Multicast Data FHR registers source with RP
2c
RP builds S,G Back Towards Source

FHR registers source with RP

2
RP builds S,G Back Towards Source
72

***When a shared MDT is already active for that group on the RP

ASM Troubleshooting:
Source Tree

R1
R1 R2
R2

FHR RP

2c
RP builds S,G Back Towards Source

2c
RP builds S,G Back Towards Source

73
ASM Troubleshooting:
Source Tree
2c RP builds S,G Back Towards Source R1
R1 R2
R2

S: 200.1.1.100 RP
Problem:
G: 239.1.1.1 FHR
(S,G) stuck in “registering” on FHR (R1)

R1#show ip mroute
(*, 239.1.1.1), 00:02:45/stopped, RP 2.2.2.2, flags: SPF
Incoming interface: Gig3/1, RPF nbr 10.1.2.2, RPF-MFD
Outgoing interface list: Null

(200.1.1.100, 239.1.1.1), 00:01:34/00:02:59, flags: PFT

Incoming interface: Gig3/21, RPF nbr 0.0.0.0, Registering (data-header), Partial-SC
Outgoing interface list: Null

74
ASM Troubleshooting:
Source Tree
2c RP builds S,G Back Towards Source R1
R1 R2
R2
Note:
Not only is the (S,G) stuck in registering, but S: 200.1.1.100 RP
G: 239.1.1.1 FHR
the CPU on FHR is also higher than normal.
Question:
Is the high CPU related to our (S,G) that is
stuck in registering?

R1#show proc cpu

CPU utilization for five seconds: 74%/31%; one minute: 73%; five
minutes: 41%
PID 5Sec 1Min 5Min TTY Process
156 42.31% 41.93% 23.57% 0 IP Input
(note: 1 mcast stream is being sent at Tx rate of 20Mbps)
75
 ASM Troubleshooting:
Source Tree
2c RP builds S,G Back Towards Source R1
R1 R2
R2
Question:
Is the high CPU related to our (S,G) that is S: 200.1.1.100 RP
stuck in registering? G: 239.1.1.1 FHR
Answer:
We actually have only 2 interfaces that are
up/up on R1. So yes, the high CPU is
related to the multicast traffic stream

There is a space after ^

R1#sh int summary | exclude ^
* GigabitEthernet3/1 0 0 0 0 0 0 20079000 2385 0
* GigabitEthernet3/21 24 96 0 0 19614000 2393 0 0 0

RXBS: rx rate (bits/sec) TXBS: tx rate (bits/sec)

RXPS: rx rate (pkts/sec) TXPS: tx rate (pkts/sec)
76
 ASM Troubleshooting:
Source Tree
2c RP builds S,G Back Towards Source R1
R1 R2
R2
Question:
S: 200.1.1.100 RP
Why is the FHR having higher CPU?
G: 239.1.1.1 FHR
Clue:
OIL on FHR for (S,G) is null
Note:
S,G tree will get built backwards from the RP.
So troubleshoot backwards from RP.

R2#show pim rpf 200.1.1.100

* 200.1.1.100/32 [-1/-1]
via Null with rpf neighbor 0.0.0.0
R2#show route 200.1.1.100
% Network not in table

77
ASM Troubleshooting:
Source Tree
Question:
Why is the FHR having higher CPU?

R1
R1 R2
R2
***When a shared MDT is already
RP active for that group on the RP
FHR
Mcast packet is encapsulated into
Mcast data unicast PIM packet:
Dest IP=RP PIM Mcast data Destination IP is the RP.
PIM header type is Register
RP sends PIM (S,G) Join Back Towards
Source if there is an active shared tree
for that G
Mcast data Once the RP sees the packets
Register Stop to the FHR come in on the (S,G), it sends a
unicast Register Stop 78
 ASM Troubleshooting:
Source Tree
2c RP builds S,G Back Towards Source R1
R1 R2
R2
Problem:
RP doesn’t know where the Source is for this S: 200.1.1.100 RP
G: 239.1.1.1 FHR
group.
Issue:
Source address not in RIB of RP –
routing problem Source
✓
Who?
Where?
RPF Nbr?
R2#show pim rpf 200.1.1.100

* 200.1.1.100/32 [-1/-1]
via Null with rpf neighbor 0.0.0.0
R2#show route 200.1.1.100
% Network not in table
79
ASM Troubleshooting:
Source Tree
Note: R1
R1 R2
R2
Fix routing problem. RP now has
Source in its RIB. S: 200.1.1.100 RP
G: 239.1.1.1 FHR
Source
✓
Who?
✓
Where?
✓ Nbr?
RPF

R2#show pim rpf 200.1.1.100 * 200.1.1.100/32 [110/2]

via Gig /0/1 with rpf neighbor 10.1.2.1

R2#show mrib route (200.1.1.100,239.1.1.1) RPF nbr: 10.1.2.1 Flags: L

Up: 00:50:36
Incoming Interface List
GigabitEthernet0/1 Flags: A, Up: 00:11:56
Outgoing Interface List
GigabitEthernet0/3 Flags: F NS, Up: 00:49:00
80
ASM Troubleshooting:
ASM Troubleshooting
Source Tree
Source Tree
2c RP builds S,G Back Towards Source R1
R1 R2
R2
Note: S: 200.1.1.100
Going back to FHR (R1) with this issue fixed FHR RP
G: 239.1.1.1
we now see that the (S,G) is no longer stuck
in registering and the CPU is no longer high.
R1#show ip mroute
(*, 239.1.1.1), 00:10:47/stopped, RP 2.2.2.2, flags: SPF
Incoming interface: GigabitEthernet3/1, RPF nbr 10.1.2.2, RPF-MFD
Outgoing interface list: Null

(200.1.1.100, 239.1.1.1), 00:10:47/00:03:20, flags: FT

Incoming interface: GigabitEthernet3/2, RPF nbr 0.0.0.0, RPF-MFD
Outgoing interface list:
GigabitEthernet3/1, Forward/Sparse, 00:03:21/00:03:12, H
81
ASM Troubleshooting:
ASM Troubleshooting
Source Tree
Source Tree
2c RP builds S,G Back Towards Source R1
R1 R2
R2

Note: S: 200.1.1.100 RP
G: 239.1.1.1 FHR
Going back to FHR (R1) with this issue
fixed we now see that the (S,G) is no
longer stuck in registering and the CPU
is no longer high.

R1#show proc cpu

CPU utilization for five seconds: 0%/0%

82
ASM Troubleshooting:
Source Tree

S: 200.1.1.100
G: 239.1.1.1
FHR Registering of Source to the RP RP
Mcast data

Potential Reasons for “Black Hole” RP Wants to Join (S,G)

1. Source not known in IGP at RP or some IR on the way back to the source
(this is the situation we just covered)
2. Usual suspects for any kind of packet loss (physical, firewall, ACL, etc)
3. Missing PIM Neighbor Relationship
4. Multicast Boundary Configured for this Multicast Group
83
ASM Troubleshooting
Source Tree
R1
R1 R2
R2

RP
S: 200.1.1.100 FHR
G: 239.1.1.1

Registering of Source to the RP

2
RP building S,G Back Towards Source

(S,G) Between RP and FHR

✓ 84
PIM-SM Frequently Forgotten Fact

85
ASM Troubleshooting
R3
R3

R1
R1 R2
R2 R5
R5

FHR RP LHR
R4

(*,G) from LHR towards the RP 1

FHR registers source with RP
2
3
RP sends PIM (S,G) Join Towards Source
LHR Cutting over to SPT (S,G)
86
ASM Troubleshooting
Joining the SPT
R3
R3

R1
R1 R2
R2 R5
R5

R4

3
LHR Cutting over to SPT (S,G)
--If SPT Threshold is not infinity (default 0) when the mcast stream comes down the shared
tree and into the LHR,
--then the LHR will try to cutover to the shortest path tree

87
ASM Troubleshooting
Joining the SPT
3 LHR Cuts over to the SPT
R3
R3

R1
R1 R2
R2 R5
R5

R4

LHR Cutting over to SPT (S,G) 3

(PIM (S,G) Joins towards FHR for source

LHR Now must answer 3 questions before Source

sending PIM (S,G) Join Who?
Where? 88

RPF Nbr?
 ASM Troubleshooting
Joining the SPT
3 LHR Cuts over to the SPT
R3
R3

R1
R1 R2
R2 R5
R5

R4

(S,G) Between LHR and FHR

✓ 3
LHR Cutting over to SPT (S,G)
89
ASM Designated Routers
Receivers
Last Hop Designated Router Responsibilities
The LH DR is responsible for sending the PIM (*,G) Join towards the RP
The LH DR will also be responsible for cutting over to the (S,G) tree

.3

Vlan

.2

LHR

90
ASM Designated Routers
Sources
First Hop Designated Router Responsibilities
The FH DR is responsible for registering the Source/Group with the RP

Router A
.3

Vlan
FHR

.2

Router B

91
ASM Troubleshooting
R3
R3

R1
R1 R2
R2 R5
R5

R4

(*,G) from LHR towards the RP 1

Registering of Source to the RP
2
3
RP sends PIM (S,G) Join Towards Source
LHR Cutting over to SPT (S,G)

92
Troubleshooting the ASM Plumbing LHR

FHR RP

93
 SSM Specific
Troubleshooting

94
SSM Specific Troubleshooting

R3
R3

R1
R1 R2
R2 R5
R5

R4

(S,G) from LHR towards Source

95
SSM Specific Troubleshooting
R3
R3 LHR receives
IGMPv3 Membership Report

ssm1
R1
R1 R2
R2 R5
R5

ssm4
R4
R2 (IR) send (S,G) Join to R1(FHR) ssm2
ssm3
LHR (R5) sends a PIM (S,G)
R4 (IR) send (S,G) Join to R2 To R4

(S,G) from LHR towards Source

96
SSM Specific Troubleshooting
ssm1 LHR Receives IGMPv3 Membership Report
LHR receives
IGMPv3 Membership Report
R3
R3
ssm1

R1
R1 R2
R2 R5
R5

R4

ssm1
LHR receives
IGMPv3 Membership Report
97
 SSM Specific Troubleshooting
ssm1 LHR Receives IGMPv3 Membership Report

R5
R5
R1 R2
ssm1
Problem: LHR receives
LHR not seeing IGMPv3 Membership Report IGMPv3 Membership Report
from Receiver

LHR#show ip mroute Shows nothing but the autorp group 224.0.1.40

LHR#show ip igmp group Is completely empty except for 224.0.1.40
LHR#show ip pim interface gig1/1
Address Interface Ver/ Nbr Query DR DR
Mode Count Intvl Prior
100.1.1.1 Gig1/1 v2/S 0 30 1 100.1.1.1
98
 SSM Specific Troubleshooting
ssm1 LHR Receives IGMPv3 Membership Report
Problem:
LHR not seeing R5
R5
IGMPv3 Membership
Report from Receiver
ssm1
Note:
Sniffer trace of IGMPv3 Membership report LHR receives
from receiver IGMPv3 Membership Report

Destination IP
224.0.0.22

IGMP v3
http://www.rfc-editor.org/rfc/rfc3376.txt 99
SSM Specific Troubleshooting
ssm1 LHR Receives IGMPv3 Membership Report
Problem:
LHR not seeing IGMPv3 R5
Membership Report
R5
from Receiver
Issue:
IGMPv3 not enabled on interface ssm1
LHR receives
Other Things to Look For:
Layer 1 issues, Layer 2 issues, Access-lists. IGMPv3 Membership Report
LHR#show ip igmp interface gig3/11 GigabitEthernet3/11 is up, line protocol is up
Internet address is 100.1.1.1/24
IGMP is enabled on interface
Current IGMP host version is 2
Current IGMP router version is 2
interface GigabitEthernet3/11
LHR#show run interface gig3/11 no switchport
ip address 100.1.1.1 255.255.255.0
ip pim sparse-mode 100
SSM Specific Troubleshooting
ssm1 LHR Receives IGMPv3 Membership Report

R5
R5
Note:
“ip igmp version 3” interface command ssm1
added under gig3/11 on the LHR
LHR receives
IGMPv3 Membership Report

LHR#show ip igmp group g3/11

IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter
232.1.1.1 Gig1/1 17:53:07 stopped 100.1.1.100
101
SSM Specific Troubleshooting
ssm2 LHR Sends (S,G) Join
R3
R3

R1
R1 R2
R2 R5
R5

R4 ssm2
LHR (R5) sends a PIM (S,G)
To R4

ssm2
LHR (R5) sends a PIM (S,G)
To R4 102
 SSM Specific Troubleshooting
R3
ssm2 LHR Sends (S,G) Join
Problem:
LHR is configured for this group R2 R5
to be an ASM group, not an SSM
group.
Mode of Group LHR
ASM R4
SSM
LHR#show ip mroute ssm Is completely empty Flags:
D - Dense,
S - Sparse,
LHR#show ip mroute B - Bidir Group
??
(*, 232.1.1.1), 17:51:47/00:02:58, RP 2.2.2.2, flags: SJC s - SSM Group
Incoming interface: GigabitEthernet3/5, RPF nbr 10.4.5.4
Outgoing interface list:
GigabitEthernet3/11, Forward/Sparse, 17:51:47/00:02:58, H
103
SSM Specific Troubleshooting
ssm2 LHR Sends (S,G) Join R3
Note:
Some platforms (XR) have the
232/8 group as SSM by default. R2 R5
In IOS it must be configured.
Mode of Group If SSM Source LHR
ASM Who? R4
SSM Where?
RPF Nbr?
LHR(config)#ip pim ssm ?
default Use 232/8 group range for SSM
range ACL for group range to be used for SSM
LHR(config)#ip pim ssm default

“232.0.0.0-232.255.255.255 (232/8) Source-Specific Multicast Block”

-- http://www.iana.org/assignments/multicast-addresses/ 104
SSM Specific Troubleshooting
ssm2 LHR Sends (S,G) Join

Question:
Assuming equal cost paths to the source from the
LHR thru R3 and R4, which router will the LHR use
as its RPF neighbor?

R3

R2 R5

R4 LHR

105
 SSM Specific Troubleshooting
R3
ssm2 LHR Sends (S,G) Join
Note:
With this group now an SSM group, the
LHR asks itself where is the Source and
R2 R5
who is the RPF nbr for this source.
Then it sends the (S,G) Join. LHR
Mode of Group If SSM Source R4
ASM ✓
Where? PIM (S,G)
✓
SSM ✓ Nbr?
RPF Join Sent
LHR#show ip mroute
(200.1.1.100, 232.1.1.1), 00:01:24/00:01:59, flags: sTI
Incoming interface: GigabitEthernet1/5, RPF nbr 10.4.5.4
Outgoing interface list:
GigabitEthernet1/1, Forward/Sparse, 00:01:24/00:01:59, H
s - SSM Group
T - SPT-bit set 106

I - Received Source Specific Host Report

SSM Specific Troubleshooting
ssm3 IR (R4) Sends (S,G) Join

R3
R3

R1
R1 R2
R2 R5
R5

ssm3 R4
R4 (IR) send (S,G) Join to R2

ssm3
R4 (IR) send (S,G) Join to R2 107
 SSM Specific Troubleshooting
ssm3 IR (R4) Sends (S,G) Join R3

Problem:
IR is configured for this group to be R2
an ASM group, not an SSM group.
R5
Mode of Group
ASM LHR
R4
SSM R4
R4#show ip mroute
??
(*, 232.1.1.1), 18:13:34/stopped, RP 2.2.2.2, flags: SP
Incoming interface: GigabitEthernet3/3, RPF nbr 10.2.4.2, Partial-SC
Outgoing interface list: Null
(200.1.1.100, 232.1.1.1), 00:00:12/00:03:17, flags:
Incoming interface: GigabitEthernet3/3, RPF nbr 10.2.4.2
Outgoing interface list:
GigabitEthernet3/5, Forward/Sparse, 00:00:12/00:03:17 108
 SSM Specific Troubleshooting
ssm3 IR (R4) Sends (S,G) Join
R3
Note:
With this group now an SSM group, the
IR (R4) asks itself where is the Source
and who is the RPF nbr for this source. R2 R5
Then it sends the (S,G) Join.
Mode of Group If SSM Source LHR
ASM ✓
Where? R4
✓
SSM ✓ Nbr?
RPF
R4
R4#show ip mroute

(200.1.1.100, 232.1.1.1), 00:05:41/00:02:44, flags: sT

Incoming interface: GigabitEthernet3/3, RPF nbr 10.2.4.2, RPF-MFD
Outgoing interface list:
GigabitEthernet3/5, Forward/Sparse, 00:05:41/00:02:44, H

109
SSM Specific Troubleshooting
ssm4 IR (R2) Sends (S,G) Join
R3
R3

R1
R1 R2
R2 R5
R5
ssm4
R4
R2 (IR) send (S,G) Join to R1(FHR)

ssm4
R2 (IR) send (S,G) Join to R1(FHR) 110
SSM Specific Troubleshooting
R3
ssm4 IR (R2) Sends (S,G) Join LHR

R1 R2 R5
Note:
XR has a very useful command
“show pim group-map”
R4

R2#show pim group-map

IP PIM Group Mapping Table
Group Range Proto Client Groups RP address Info

224.0.1.39/32* DM perm 0 0.0.0.0

224.0.1.40/32* DM perm 1 0.0.0.0
224.0.0.0/24* NO perm 0 0.0.0.0
232.0.0.0/8* SSM config 1 0.0.0.0
224.0.0.0/4* SM config 0 2.2.2.2 RPF: De0,2.2.2.2
224.0.0.0/4 SM static 0 0.0.0.0 RPF: Null,0.0.0.0
111
 SSM Specific Troubleshooting R3
ssm4 IR (R2) Sends (S,G) Join LHR
Note: R1 R2 R5
IR (R2) asks itself where is the Source
and who is the RPF nbr for this source.
Then it sends the (S,G) Join. R4
Mode of Group If SSM Source
✓ R2 sends
ASM Where?
✓
SSM ✓ Nbr?
RPF PIM (S,G) Join

R2#show mrib ipv4 route 232.1.1.1

(200.1.1.100,232.1.1.1) RPF nbr: 10.1.2.1 Flags:
Up: 01:22:22
Incoming Interface List
GigabitEthernet0/1/0/1 Flags: A, Up: 01:22:22
Outgoing Interface List
GigabitEthernet0/1/0/3 Flags: F NS, Up: 01:22:22
112
SSM Specific Troubleshooting R3
ssm4 IR (R2) Sends (S,G) Join LHR

Note: R1 R2 R5
(S,G) MDT built all the way from the
LHR to the FHR. But note how no
R4
traffic has yet been sent from the
source. R2 sends
PIM (S,G) Join
R1#show ip mroute 232.1.1.1
(200.1.1.100, 232.1.1.1), 00:00:28/00:03:01, flags: sT
Incoming interface: GigabitEthernet3/21, RPF nbr 0.0.0.0, RPF-MFD
Outgoing interface list:
GigabitEthernet3/1, Forward/Sparse, 00:00:28/00:03:01, H
R1#show ip mroute 232.1.1.1 count
Group: 232.1.1.1, Source count: 1, Pkts forwarded: 0, Pkts received: 0
Source: 200.1.1.100/32, Forwarding: 0/0/0/0, Other: 0/0/0
113
SSM Specific Troubleshooting
R3
R3
ssm1
LHR receives
R1
R1 R2
R2 IGMPv3 Membership
R5R5 Report

ssm4 R4
R2 (IR) send (S,G) Join to R1(FHR) ssm2
ssm3
LHR (R5) sends a PIM (*,G)
R4 (IR) send (S,G) Join to R2 To R4

(S,G) from LHR towards Source

114
Troubleshooting with Fish

115
NO Receivers Getting Mcast

FHR RP
LHR
Start at RP for this Group
 *,G w/ OIL?
• NO? Pick 1 LHR and start there
 S,G
• NO? Go to FHR

116
Some Receivers Getting Mcast

FHR RP
LHR

Start at LHR of Broken Receiver

 Pull?
 Who, Where, What

117
 SSM Troubleshooting with Fish LHR

FHR

Start at FHR
 S,G w/ OIL?
• NO? Pick 1 LHR and start there
• YES? Check to see if traffic coming in from Source

118
Troubleshooting Toolbox
Facts to Remember

• Multicast Trees get built backwards towards the root

• Multicast traffic in ASM and SSM are triggered via a “pull”

• High on the “food chain” dependent on routing working and PIM working

119
Troubleshooting Toolbox
Questions to Remember

• Who is the root?

• Where is the root?

• What is the PIM RPF neighbor towards the root?

120
Troubleshooting Toolbox
Checklist

 IGMP Membership Report Received by LHR

 “WHO:” is the root?
 “WHERE” is the root?
 “WHAT” is the PIM RPF neighbor towards the root?

121
Questions?

122
Participate in the “My Favorite Speaker” Contest
Promote Your Favorite Speaker and You Could be a Winner
• Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)
• Send a tweet and include
• Your favorite speaker’s Twitter handle <@DeniseFishburne>
• Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers
• Don’t forget to follow @CiscoLive and @CiscoPress
• View the official rules at http://bit.ly/CLUSwin

123
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
• Complete your session surveys
though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online

124
Continue Your Education
• Demos in the Cisco Campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings

125
Thank you

126