Multicast Troubleshooting

Transcription

1

2 Multicast Troubleshooting Denise Fish Fishburne Customer Proof of Concept Team Lead CCIE #2639, CCDE 2009:0014 BRKIPM-2264

3 Cisco Spark How Questions? Use Cisco Spark to chat with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space Cisco Spark spaces will be available until July 3, cs.co/ciscolivebot#brkipm Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Agenda Quick Review Multicast Troubleshooting Methodology Troubleshooting ASM Troubleshooting SSM Final Troubleshooting Takeaways

5 Housekeeping Items Cell Phones Intermediate Class Assumes Multicast Operational Experience Questions BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Quick Review

7 Quick Review Agenda Frequently Used Terms Tree Growth Sending a (*,G) PIM Join Registering a Source Designated Routers

8 Frequently Used Terms ASM - Any Source Multicast SSM MDT FHR LHR IR DR IGMP - Source Specific Multicast - Multicast Distribution Tree - Rendezvous Point - First Hop Router - Last Hop Router - Intermediate Router - Designated Router - Internet Group Management Protocol BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Frequently Used Terms Any Source Multicast (ASM) Original (Classic) PIM-SM Supports both Shared and Source Trees Source Specific Multicast (SSM) a.k.a. Single Source Multicast Supports only Source Trees BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 Frequently Used Terms (*,G) = *,G = Shared Tree = tree *,G MDT LHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Frequently Used Terms (S,G) = S,G = Shortest Path Tree = Source tree S,G MDT FHR LHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Frequently Used Terms (*,G) = *,G = Shared Tree = tree (S,G) = S,G = Source Tree = Shortest Path Tree Q: What is the Root of the (*,G) Tree? A: Q: What is the Root of the (S,G) Tree? A: Source BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Frequently Used Terms IIF = Incoming Interface Interface towards the root of the tree OIL = Outgoing interface List Interfaces that have received PIM Joins or IGMP membership requests Q: What does the IIF of the (*,G) Tree point towards? A: Q: What does the IIF of the (S,G) Tree point towards A: Source BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Tree Growth Nature s Way In Nature, trees grow UP and OUT from the ROOT BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Tree Growth Multicast Distribution Tree (MDT) With Multicast, trees grow TOWARD the ROOT BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Sending a (*,G) PIM Join Router A PIM Join Router B Question: When Router A sends a PIM (*,G) Join out, what will be the Destination IP? IP address of the? ? (All PIM Routers) IP of Router B ( ) **Assume Router A knows the, has the in its RIB via an IGP with B and has a PIM neighbor with Router B BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Sending a (*,G) PIM Join The PIM Header PIM header common to all PIM messages: Some PIM Types are: Message Type Destination 0 = Hello Multicast to ALL-PIM-ROUTERS 1 = Register Unicast to 2 = Register-Stop Unicast to source of Register packet 3 = Join/Prune Multicast to ALL-PIM-ROUTERS 4 = Bootstrap Multicast to ALL-PIM-ROUTERS 5 = Assert Multicast to ALL-PIM-ROUTERS BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Sending a (*,G) PIM Join Router A.5.4 DIP PIM Join Router B Q: When Router A sends a PIM (*,G) Join out, what will be the Destination IP? IP of the? ? (All PIM Routers) IP of Router B ( ) BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Sending a (*,G) PIM Join A B DIP PIM Join.1 C Question: If the destination IP address of the Join is (all PIM Routers), then how do we keep Router B and Router C from BOTH acting on that join? BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Sending a (*,G) Join A B DIP PIM Join.1 Answer: The address of the upstream neighbor (B or C) is the target of the message C BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Sending a (*,G) PIM Join A B PIM Join.1 Destination IP C IP address of B BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Registering a Source*** Mcast Data Mcast data Mcast data Mcast data gig0 added to the OIL for (S,G) FHR Dest IP= gig0 PIM Register Stop to the FHR Mcast data Mcast data ***When a shared MDT is already active for that group on the = Mcast packet is encapsulated into unicast PIM packet: Destination IP is the. PIM header type is Register sends PIM (S,G) Join Back Towards Source if there is an active shared tree for that G Mcast packets now get sent out twice 1.With Register 2.Out (S,G) Once the sees the packets come in on the (S,G),. it sends a unicast Register Stop Mcast packets now only out the (S,G) tree BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Designated Routers (DRs).3 Router A Vlan.2 FHR Router B Q: Which router should register the source with the? A: The FHR that is the DR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Designated Routers (DRs) Q: Which router should send the PIM (*,G) Join? A: The LHR that is the DR.3.2 LHR Vlan BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Multicast Troubleshooting Methodology

26 Multicast Troubleshooting Methodology Agenda The Plumbing Troubleshooting Toolbox

27 Building the Plumbing Unicast Ping EIG EIG EIG advertise advertise advertise / / /24 Subnet / Point A Point B BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Building the Plumbing Multicast*** Source (Tx) PIM PIM PIM FHR PIM (*,G) PIM (*,G) LHR PIM Unicast Register Join PIM Join Join PIM (S,G) Join PIM (S,G) Join ***In this example, LHR is configured to not cut over to the S,G tree. BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 In solving a problem of this sort, the grand thing is to be able to reason backward. That is a very useful accomplishment, and a very easy one, but people do not practice it much. Sherlock Holmes BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 Troubleshooting the ASM Plumbing LHR FHR MDTs Built Backwards Towards Root Traffic Troubleshooting BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 Troubleshooting the SSM Plumbing LHR FHR MDTs Built Backwards Towards Root Traffic Troubleshooting BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 Troubleshooting Toolbox Go To Commands show ip mroute show ip mroute count show ip pim neighbor show ip pim interface show ip igmp group show ip pim rp [group] show ip pim rp mapping [group] show ip rpf [address] BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 Troubleshooting Toolbox Facts to Remember Multicast Trees get built backwards towards the root Multicast traffic in ASM and SSM are triggered via a pull High on the food chain dependent on routing working and PIM working BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Troubleshooting Toolbox Questions to Remember Who is the root? Where is the root? What is the PIM F neighbor towards the root? BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 Troubleshooting Toolbox Checklist IGMP Membership Report Received by LHR WHO: is the root? WHERE is the root? WHAT is the PIM F neighbor towards the root? BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 ASM Troubleshooting

37 ASM Troubleshooting R3 R3 R5 R5 FHR R4 LHR (*,G) from LHR towards the 1 2 FHR registers source with sends PIM (S,G) Join Towards Source 3 LHR Cutting over to SPT (S,G) BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 ASM Troubleshooting Shared Tree R3 R3 R5 R5 FHR R4 LHR 2 (*,G) from LHR towards the 1 FHR registers source with sends PIM (S,G) Join Towards Source LHR Cutting over to SPT (S,G) 3 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 ASM Troubleshooting: Shared Tree 1c R4 (an IR) sends a PIM (*,G) To () R3 R4 1b LHR R5 1a LHR receives IGMP Membership Report LHR (R5) sends a PIM (*,G) To R4 (*,G) from LHR towards the 1 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 ASM Troubleshooting: Shared Tree R3 LHR R5 R4 IGMP Membership Report Received by LHR WHO: LHR knows who the is for this group WHERE: LHR knows where (RIB) the is WHAT: LHR has a PIM Neighbor to send the PIM (*,G) Join to 1a LHR receives IGMP Membership Report BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 ASM Troubleshooting: Shared Tree 1a LHR Receives IGMP Membership Report LHR R5 IGMP Membership Report Received by LHR LHR#show ip mroute LHR#show ip igmp group LHR#show ip igmp interface Is completely empty Is completely empty Is completely empty BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 ASM Troubleshooting: Shared Tree 1a LHR Receives IGMP Membership Report LHR IGMP Membership Report Received by LHR PIM not enable on interface of Receiver Layer 1 issues Layer 2 issues Access-List IGMP membership report not being sent R5 LHR#show run interface gig1/1 interface GigabitEthernet1/1 no switchport ip address ip pim sparse-mode BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 ASM Troubleshooting: Shared Tree 1a LHR Receives IGMP Membership Report LHR R5 IGMP Membership Report Received by LHR LHR#show run interface gig1/1 LHR#show ip igmp groups IGMP Connected Group Membership interface GigabitEthernet1/1 no switchport ip address ip pim sparse-mode Group Address Interface Uptime Expires Last Reporter GigabitEthernet1/1 00:18:35 00:02: BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 LHR R5 WHO: LHR knows who the is for this group WHERE: LHR knows where (RIB) the is WHAT: LHR has a PIM Neighbor to send the PIM (*,G) Join to R4 1b LHR (R5) sends a PIM (*,G) To R4 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 R5 Who? Where? F Nbr? R4 LHR LHR#show ip mroute (*, ), 00:01:23/00:02:48, , flags: SJC Incoming interface: Null, F nbr ? Outgoing interface list: GigabitEthernet1/1, Forward/Sparse, 00:01:23/00:02:48, H LHR#show ip pim rp Group: , : ? BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 Note: information can be shared 3 ways Static Auto-rp BSR R4 R5 LHR LHR(config)#ip pim rp-address ? <1-99> Access-list reference for group < > Access-list reference for group (expanded range) WORD IP Named Standard Access list override Overrides dynamically learnt mappings BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 R5 Who? Where? F Nbr? R4 LHR LHR#sh ip pim rp Group: , : , uptime 00:00:39, expires never LHR#sh ip pim rp mapping PIM Group-to- Mappings Group(s): /4, Static-Override : (?) BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 Who? Where? F Nbr? R4 R5 LHR LHR#show ip mroute LHR#show ip rpf (*, ), 00:08:18/00:02:46, , flags: SJC Incoming interface: Null, F nbr Outgoing interface list: GigabitEthernet1/1, Forward/Sparse, 00:08:18/00:02:46, H F information for? ( ) failed, no route exists LHR#show ip route % Network not in table BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 Who? Where? F Nbr? R4 R5 LHR LHR#show ip route /32 is subnetted, 1 subnets O [110/42] via , Gig 1/5 [110/42] via , Gig 1/4 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 Who? Where? F Nbr? R4 R5 LHR LHR#show ip mroute (*, ), 00:08:18/00:02:46, , flags: SJC Incoming interface: Null, F nbr Outgoing interface list: GigabitEthernet1/1, Forward/Sparse, 00:08:18/00:02:46, H BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 Who? Where? F Nbr? R4 R5 LHR LHR#show ip rpf LHR#show ip pim neighbor F information for? ( ) failed, no route exists No PIM neighbors listed BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 R4 R5 LHR LHR#show ip pim neighbor PIM Neighbor Table Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority, P - Proxy Capable, S - State Refresh Capable Neighbor Interface Uptime/Expires GigabitEthernet1/4 00:17:57/00:01: GigabitEthernet1/5 00:17:56/00:01:31 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 Question: Why was chosen as the F neighbor? R5 Answer: If ECMP are available, the F for the multicast traffic will be based on highest IP address for the F neighbor R4 LHR LHR#show ip rpf F information for? ( ) F interface: GigabitEthernet1/5 F neighbor:? ( ) F route/mask: /32 F type: unicast (ospf 1) BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 ASM Troubleshooting: Shared Tree 1b LHR Sends PIM (*,G) Join R3 Note: LHR now has all the information it needs to send a PIM (*,G) join to its F neighbor. R5 Who? Where? F Nbr? LHR#show ip mroute *,G IIF OIL (*, ), 00:36:39/00:02:56, , flags: SJC Incoming interface: Gig 1/5, F nbr Outgoing interface list: Gig 1/1, Forward/Sparse-Dense, 00:08:08/00:02:56, H R4 LHR 1b LHR (R5) sends a PIM (*,G) To R4 flags F nbr BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 ASM Troubleshooting: Shared Tree 1c IR Sends PIM (*,G) Join R3 R5 R4 (an IR) sends a PIM (*,G) To () R4 1c R4 LHR WHO: IR knows who the is for this group WHERE: IR knows where (RIB) the is WHAT: IR has a PIM Neighbor to send the PIM (*,G) Join to BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 ASM Troubleshooting: Shared Tree 1c IR Sends PIM (*,G) Join R3 Note: Telnetting backwards towards the root is a good way to troubleshoot. R5 LHR#show ip mroute (*, ), 00:36:39/00:02:56, , flags: SJC Incoming interface: GigabitEthernet1/5, F nbr Outgoing interface list: Gig 1/1, Forward/Sparse-Dense, 00:08:08/00:02:56, H LHR#telnet Trying Open R4#show ip mroute (*, ), 00:48:49/00:02:33, , flags: S Incoming interface: Gig 3/3, F nbr , F-MFD Outgoing interface list: Gig 3/5, Forward/Sparse, 00:47:16/00:02:33, H R4 BRKIPM-2264 R4 LHR Who? Where? F Nbr? 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 ASM Troubleshooting: Shared Tree 1c IR Sends PIM (*,G) Join R3 Note: R4 had all the information it needed to send a PIM (*,G) join to its F neighbor. R4 R4#telnet R4 (an IR) sent a PIM (*,G) R4 Trying Open To () 1c #show mrib route (*, ) F nbr: Flags: C Up: 00:52:12 Incoming Interface List Decapstunnel0 Flags: A, Up: 00:52:12 Outgoing Interface List GigabitEthernet0/0/0/3 Flags: F NS, Up: 00:52:12 R5 LHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 ASM Troubleshooting: Shared Tree R3 LHR R4 (an IR) sends a PIM (*,G) To () (*,G) Between and LHR 1c R4 1b R5 (*,G) from LHR towards the 1 1a LHR receives IGMP Membership Report LHR (R5) sends a PIM (*,G) To R4 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 ASM Troubleshooting R3 R3 R5 R5 FHR R4 LHR 2 (*,G) from LHR towards the 1 FHR registers source with sends PIM (S,G) Join Towards Source LHR Cutting over to SPT (S,G) 3 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 ASM Troubleshooting: Source Tree 2a FHR receives Multicast Data FHR 2b FHR registers source with builds S,G Back Towards Source 2c 2 FHR registers source with builds S,G Back Towards Source ***When a shared MDT is already active for that group on the BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 ASM Troubleshooting: Source Tree 2a FHR receives Multicast Data FHR 2b FHR registers source with builds S,G Back Towards Source 2c 2 FHR registers source with builds S,G Back Towards Source ***When a shared MDT is already active for that group on the BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 ASM Troubleshooting: Source Tree 2a FHR Receives Multicast Data 2a FHR receives Multicast Data FHR 2a FHR receives Multicast Data BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 ASM Troubleshooting: Source Tree 2a FHR Receives Multicast Data FHR Receives Multicast Data PIM not enable on interface of Receiver Layer 1 issues Layer 2 issues Access-List IGMP membership report not being sent S: G: a FHR receives Multicast Data FHR #show ip mroute Empty #show run interface gig3/2 interface GigabitEthernet3/2 no switchport ip address ip pim sparse-mode BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 ASM Troubleshooting: Source Tree 2a FHR Receives Multicast Data Note: We see that a (*,G) for that group has been built. S: G: FHR #show ip mroute (*, ), 00:02:17/00:00:42, , flags: SP Incoming interface: Null, F nbr Outgoing interface list: Null BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 ASM Troubleshooting: Source Tree 2a FHR receives Multicast Data FHR 2b FHR registers source with builds S,G Back Towards Source 2c 2 FHR registers source with builds S,G Back Towards Source ***When a shared MDT is already active for that group on the BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 ASM Troubleshooting: Source Tree 2b FHR Registers Source with FHR 2b FHR registers source with 2b FHR registers source with BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 ASM Troubleshooting: Source Tree 2b FHR Registers Source with Note: We see that a (*,G) for that group has been built. S: G: FHR Question: Why is there no (S,G)? #show ip mroute (*, ), 00:02:17/00:00:42, , flags: SP Incoming interface: Null, F nbr Outgoing interface list: Null BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 ASM Troubleshooting: Source Tree 2b FHR Registers Source with Note: We see that a (*,G) for that group has been built. S: G: FHR Question: Why is there no (S,G)? #show ip mroute count IP Multicast Statistics 2 routes using 1230 bytes of memory 2 groups, 0.00 average sources per group Forwarding Counts: Pkt Count/Pkts per second/avg Pkt Size/Kilobits per second Other counts: Total/F failed/other drops(oif-null, rate-limit etc) Group: , Source count: 0, Packets forwarded: 0, Packets received: tree: Forwarding: 0/0/0/0, Other: 6027/6026/1 F failed BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 ASM Troubleshooting: Source Tree 2b FHR Registers Source with Problem: FHR does not know who the is for this group. S: G: FHR Who? Where? F Nbr? #show ip mroute (*, ), 00:02:17/00:00:42, , flags: SP Incoming interface: Null, F nbr Outgoing interface list: Null #sh ip rpf F information for? ( ) failed, no route exists BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 ASM Troubleshooting: Source Tree 2b FHR Registers Source with Note: information can be shared 3 ways Static Auto-rp BSR Who? Where? F Nbr? ip pim rp-address override #show ip mroute S: G: FHR #sh ip pim rp mapping PIM Group-to- Mappings Group(s): /4, Static-Override : (?) (*, ), 00:02:17/00:00:42, , flags: SP Incoming interface: Null, F nbr Outgoing interface list: Null BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 ASM Troubleshooting: Source Tree 2b FHR Registers Source with Problem: FHR doesn t know who the F neighbor is for the shared tree for this group. Who? Where? F Nbr? S: G: FHR #show ip route #show ip rpf /32 is subnetted, 1 subnets O [110/2] via , 01:11:17, Gig3/1 F information for? ( ) failed, no route exists #show ip pim neighbor No PIM neighbors BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 ASM Troubleshooting: Source Tree 2b FHR Registers Source with Note: ip pim sparse-mode added to Gig3/1 on FHR () S: G: FHR #show ip mroute (*, ), 00:02:45/stopped, , flags: SPF Incoming interface: Gig3/1, F nbr , F-MFD Outgoing interface list: Null ( , ), 00:01:34/00:02:59, flags: PFT Incoming interface: Gig3/21, F nbr , Registering (data-header), Partial-SC Outgoing interface list: Null BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 ASM Troubleshooting: Source Tree 2a FHR receives Multicast Data FHR 2b FHR registers source with builds S,G Back Towards Source 2c 2 FHR registers source with builds S,G Back Towards Source ***When a shared MDT is already active for that group on the BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 ASM Troubleshooting: Source Tree FHR builds S,G Back Towards Source 2c builds S,G Back Towards Source 2c BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 ASM Troubleshooting: Source Tree 2c builds S,G Back Towards Source Problem: (S,G) stuck in registering on FHR () S: G: FHR #show ip mroute (*, ), 00:02:45/stopped, , flags: SPF Incoming interface: Gig3/1, F nbr , F-MFD Outgoing interface list: Null ( , ), 00:01:34/00:02:59, flags: PFT Incoming interface: Gig3/21, F nbr , Registering (data-header), Partial-SC Outgoing interface list: Null BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 ASM Troubleshooting: Source Tree 2c builds S,G Back Towards Source Note: Not only is the (S,G) stuck in registering, but the CPU on FHR is also higher than normal. Question: Is the high CPU related to our (S,G) that is stuck in registering? S: G: FHR #show proc cpu CPU utilization for five seconds: 74%/31%; one minute: 73%; five minutes: 41% PID 5Sec 1Min 5Min TTY Process % 41.93% 23.57% 0 IP Input (note: 1 mcast stream is being sent at Tx rate of 20Mbps) BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 ASM Troubleshooting: Source Tree 2c builds S,G Back Towards Source Question: Is the high CPU related to our (S,G) that is stuck in registering? Answer: We actually have only 2 interfaces that are up/up on. So yes, the high CPU is related to the multicast traffic stream S: G: There is a space after ^ FHR #sh int summary exclude ^ * GigabitEthernet3/ * GigabitEthernet3/ RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec) TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec) BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 ASM Troubleshooting: Source Tree 2c builds S,G Back Towards Source Question: Why is the FHR having higher CPU? Clue: OIL on FHR for (S,G) is null Note: S,G tree will get built backwards from the. So troubleshoot backwards from. #show pim rpf S: G: FHR * /32 [-1/-1] via Null with rpf neighbor #show route % Network not in table BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 ASM Troubleshooting: Source Tree Question: Why is the FHR having higher CPU? Mcast data Mcast data FHR Dest IP= PIM Mcast data Register Stop to the FHR ***When a shared MDT is already active for that group on the Mcast packet is encapsulated into unicast PIM packet: Destination IP is the. PIM header type is Register sends PIM (S,G) Join Back Towards Source if there is an active shared tree for that G Once the sees the packets come in on the (S,G), it sends a unicast Register Stop BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 ASM Troubleshooting: Source Tree 2c builds S,G Back Towards Source Problem: doesn t know where the Source is for this group. Issue: Source address not in RIB of routing problem Source Who? Where? F Nbr? #show pim rpf S: G: FHR * /32 [-1/-1] via Null with rpf neighbor #show route % Network not in table BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 ASM Troubleshooting: Source Tree Note: Fix routing problem. now has Source in its RIB. Source Who? Where? F Nbr? S: G: FHR #show pim rpf * /32 [110/2] via Gig /0/1 with rpf neighbor #show mrib route ( , ) F nbr: Flags: L Up: 00:50:36 Incoming Interface List GigabitEthernet0/1 Flags: A, Up: 00:11:56 Outgoing Interface List GigabitEthernet0/3 Flags: F NS, Up: 00:49:00 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 ASM Troubleshooting: Source Tree Source Tree 2c builds S,G Back Towards Source Note: Going back to FHR () with this issue fixed we now see that the (S,G) is no longer stuck in registering and the CPU is no longer high. #show ip mroute S: G: FHR (*, ), 00:10:47/stopped, , flags: SPF Incoming interface: GigabitEthernet3/1, F nbr , F-MFD Outgoing interface list: Null ( , ), 00:10:47/00:03:20, flags: FT Incoming interface: GigabitEthernet3/2, F nbr , F-MFD Outgoing interface list: GigabitEthernet3/1, Forward/Sparse, 00:03:21/00:03:12, H BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 ASM Troubleshooting: Source Tree Source Tree 2c builds S,G Back Towards Source Note: Going back to FHR () with this issue fixed we now see that the (S,G) is no longer stuck in registering and the CPU is no longer high. S: G: FHR #show proc cpu CPU utilization for five seconds: 0%/0% BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 83

84 ASM Troubleshooting: Source Tree S: G: Mcast data FHR Registering of Source to the Potential Reasons for Black Hole 1. Source not known in IGP at or some IR on the way back to the source (this is the situation we just covered) 2. Usual suspects for any kind of packet loss (physical, firewall, ACL, etc) 3. Missing PIM Neighbor Relationship 4. Multicast Boundary Configured for this Multicast Group Wants to Join (S,G) BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 ASM Troubleshooting: Source Tree S: G: FHR 2 Registering of Source to the building S,G Back Towards Source (S,G) Between and FHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 85

86 The default behavior of PIM-SM is that routers with directly connected members will join the shortest path tree as soon as they detect a new multicast source. PIM-SM Frequently Forgotten Fact BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 86

87 ASM Troubleshooting R3 R3 R5 R5 FHR R4 LHR 2 (*,G) from LHR towards the 1 FHR registers source with sends PIM (S,G) Join Towards Source LHR Cutting over to SPT (S,G) 3 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 87

88 ASM Troubleshooting: Joining the SPT R3 R3 R5 R5 LHR Cutting over to SPT (S,G) --If SPT Threshold is not infinity (default 0) when the mcast stream comes down the shared tree and into the LHR, --then the LHR will try to cutover to the shortest path tree R4 3 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 88

89 ASM Troubleshooting: Joining the SPT 3 LHR Cuts over to the SPT R3 R3 R5 R5 R4 LHR Cutting over to SPT (S,G) (PIM (S,G) Joins towards FHR for source 3 LHR Now must answer 3 questions before sending PIM (S,G) Join Source Who? Where? F Nbr? BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 89

90 ASM Troubleshooting: Joining the SPT 3 LHR Cuts over to the SPT R3 R3 R5 R5 R4 (S,G) Between LHR and FHR LHR Cutting over to SPT (S,G) 3 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 90

91 ASM Designated Routers: Receivers Last Hop Designated Router Responsibilities The LH DR is responsible for sending the PIM (*,G) Join towards the The LH DR will also be responsible for cutting over to the (S,G) tree.3 Vlan.2 LHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 91

92 ASM Designated Routers: Sources First Hop Designated Router Responsibilities The FH DR is responsible for registering the Source/Group with the.3 Router A Vlan FHR.2 Router B BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 92

93 ASM Troubleshooting R3 R3 R5 R5 R4 2 (*,G) from LHR towards the 1 Registering of Source to the sends PIM (S,G) Join Towards Source LHR Cutting over to SPT (S,G) 3 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 93

94 Troubleshooting the ASM Plumbing LHR FHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 94

95 SSM Specific Troubleshooting

96 SSM Specific Troubleshooting R3 R3 R5 R5 R4 (S,G) from LHR towards Source BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 96

97 SSM Specific Troubleshooting R3 R3 LHR receives IGMPv3 Membership Report R5 R5 ssm1 ssm4 (IR) send (S,G) Join to (FHR) ssm3 R4 (IR) send (S,G) Join to R4 ssm2 LHR (R5) sends a PIM (S,G) To R4 (S,G) from LHR towards Source BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 97

98 SSM Specific Troubleshooting ssm1 LHR Receives IGMPv3 Membership Report R3 R3 LHR receives IGMPv3 Membership Report ssm1 R5 R5 R4 ssm1 LHR receives IGMPv3 Membership Report BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 98

99 SSM Specific Troubleshooting ssm1 LHR Receives IGMPv3 Membership Report Problem: LHR not seeing IGMPv3 Membership Report from Receiver R5R5 ssm1 LHR receives IGMPv3 Membership Report LHR#show ip mroute Shows nothing but the autorp group LHR#show ip igmp group Is completely empty except for LHR#show ip pim interface gig1/1 Address Interface Ver/ Nbr Query DR DR Mode Count Intvl Prior Gig1/1 v2/s BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 99

100 SSM Specific Troubleshooting ssm1 Problem: LHR not seeing IGMPv3 Membership Report from Receiver LHR Receives IGMPv3 Membership Report Note: Sniffer trace of IGMPv3 Membership report from receiver R5R5 ssm1 LHR receives IGMPv3 Membership Report Destination IP IGMP v3 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 100

101 SSM Specific Troubleshooting ssm1 LHR Receives IGMPv3 Membership Report Problem: LHR not seeing IGMPv3 Membership Report from Receiver Issue: IGMPv3 not enabled on interface Other Things to Look For: Layer 1 issues, Layer 2 issues, Access-lists. LHR#show run interface gig3/11 R5R5 ssm1 LHR receives IGMPv3 Membership Report LHR#show ip igmp interface gig3/11 GigabitEthernet3/11 is up, line protocol is up Internet address is /24 IGMP is enabled on interface Current IGMP host version is 2 Current IGMP router version is 2 interface GigabitEthernet3/11 no switchport ip address ip pim sparse-mode BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 101

102 SSM Specific Troubleshooting ssm1 LHR Receives IGMPv3 Membership Report R5R5 Note: ip igmp version 3 interface command added under gig3/11 on the LHR ssm1 LHR receives IGMPv3 Membership Report LHR#show ip igmp group g3/11 IGMP Connected Group Membership Group Address Interface Uptime Expires Last Reporter Gig1/1 17:53:07 stopped BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 102

103 SSM Specific Troubleshooting ssm2 LHR Sends (S,G) Join R3 R3 R5 R5 R4 ssm2 LHR (R5) sends a PIM (S,G) To R4 ssm2 LHR (R5) sends a PIM (S,G) To R4 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 103

104 SSM Specific Troubleshooting ssm2 LHR Sends (S,G) Join Problem: LHR is configured for this group to be an ASM group, not an SSM R5 group. Mode of Group ASM R4 LHR SSM LHR#show ip mroute ssm Is completely empty Flags: D - Dense, S - Sparse, LHR#show ip mroute B - Bidir Group (*, ), 17:51:47/00:02:58, ,?? flags: SJC s - SSM Group Incoming interface: GigabitEthernet3/5, F nbr Outgoing interface list: GigabitEthernet3/11, Forward/Sparse, 17:51:47/00:02:58, H R3 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 104

105 SSM Specific Troubleshooting ssm2 LHR Sends (S,G) Join Note: Some platforms (XR) have the 232/8 group as SSM by default. In IOS it must be configured. Mode of Group ASM SSM If SSM Source Who? Where? F Nbr? LHR(config)#ip pim ssm? default Use 232/8 group range for SSM range ACL for group range to be used for SSM LHR(config)#ip pim ssm default R3 R4 R5 LHR (232/8) Source-Specific Multicast Block -- BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 105

106 SSM Specific Troubleshooting ssm2 LHR Sends (S,G) Join Question: Assuming equal cost paths to the source from the LHR thru R3 and R4, which router will the LHR use as its F neighbor? R3 R5 R4 LHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 106

107 SSM Specific Troubleshooting ssm2 LHR Sends (S,G) Join Note: With this group now an SSM group, the LHR asks itself where is the Source and who is the F nbr for this source. Then it sends the (S,G) Join. Mode of Group If SSM Source ASM Where? SSM F Nbr? LHR#show ip mroute ( , ), 00:01:24/00:01:59, flags: sti Incoming interface: GigabitEthernet1/5, F nbr Outgoing interface list: GigabitEthernet1/1, Forward/Sparse, 00:01:24/00:01:59, H s - SSM Group T - SPT-bit set I - Received Source Specific Host Report R3 R4 R5 LHR PIM (S,G) Join Sent BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 107

108 SSM Specific Troubleshooting ssm3 IR (R4) Sends (S,G) Join R3 R3 R5 R5 ssm3 R4 (IR) send (S,G) Join to R4 ssm3 R4 (IR) send (S,G) Join to BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 108

109 SSM Specific Troubleshooting ssm3 IR (R4) Sends (S,G) Join Problem: IR is configured for this group to be an ASM group, not an SSM group. Mode of Group ASM SSM R4#show ip mroute?? (*, ), 18:13:34/stopped, , flags: SP Incoming interface: GigabitEthernet3/3, F nbr , Partial-SC Outgoing interface list: Null ( , ), 00:00:12/00:03:17, flags: Incoming interface: GigabitEthernet3/3, F nbr Outgoing interface list: GigabitEthernet3/5, Forward/Sparse, 00:00:12/00:03:17 R4 R3 R4 R5 LHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 109

110 SSM Specific Troubleshooting ssm3 IR (R4) Sends (S,G) Join Note: With this group now an SSM group, the IR (R4) asks itself where is the Source and who is the F nbr for this source. Then it sends the (S,G) Join. Mode of Group ASM SSM R4#show ip mroute If SSM Source Where? F Nbr? R4 R3 R4 R5 LHR ( , ), 00:05:41/00:02:44, flags: st Incoming interface: GigabitEthernet3/3, F nbr , F-MFD Outgoing interface list: GigabitEthernet3/5, Forward/Sparse, 00:05:41/00:02:44, H BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 110

111 SSM Specific Troubleshooting ssm4 IR () Sends (S,G) Join R3 R3 R5 R5 ssm4 (IR) send (S,G) Join to (FHR) R4 ssm4 (IR) send (S,G) Join to (FHR) BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 111

112 SSM Specific Troubleshooting ssm4 IR () Sends (S,G) Join R3 LHR Note: XR has a very useful command show pim group-map R4 R5 #show pim group-map IP PIM Group Mapping Table Group Range Proto Client Groups address Info /32* DM perm /32* DM perm /24* NO perm /8* SSM config /4* SM config F: De0, /4 SM static F: Null, BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 112

113 SSM Specific Troubleshooting ssm4 IR () Sends (S,G) Join Note: IR () asks itself where is the Source and who is the F nbr for this source. Then it sends the (S,G) Join. Mode of Group If SSM Source ASM Where? SSM F Nbr? sends PIM (S,G) Join R3 R4 LHR R5 #show mrib ipv4 route ( , ) F nbr: Flags: Up: 01:22:22 Incoming Interface List GigabitEthernet0/1/0/1 Flags: A, Up: 01:22:22 Outgoing Interface List GigabitEthernet0/1/0/3 Flags: F NS, Up: 01:22:22 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 113

114 SSM Specific Troubleshooting ssm4 IR () Sends (S,G) Join Note: (S,G) MDT built all the way from the LHR to the FHR. But note how no traffic has yet been sent from the source. #show ip mroute ( , ), 00:00:28/00:03:01, flags: st Incoming interface: GigabitEthernet3/21, F nbr , F-MFD Outgoing interface list: GigabitEthernet3/1, Forward/Sparse, 00:00:28/00:03:01, H #show ip mroute count sends PIM (S,G) Join Group: , Source count: 1, Pkts forwarded: 0, Pkts received: 0 Source: /32, Forwarding: 0/0/0/0, Other: 0/0/0 R3 R4 LHR R5 BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 114

115 SSM Specific Troubleshooting R3 R3 ssm1 LHR receives IGMPv3 Membership Report R5 R5 ssm4 (IR) send (S,G) Join to (FHR) ssm3 R4 (IR) send (S,G) Join to R4 ssm2 LHR (R5) sends a PIM (*,G) To R4 (S,G) from LHR towards Source BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 115

116 Final Troubleshooting Takeaways

117 Troubleshooting Toolbox Facts to Remember Multicast Trees get built backwards towards the root Multicast traffic in ASM and SSM are triggered via a pull High on the food chain dependent on routing working and PIM working BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 117

118 Troubleshooting Toolbox Questions to Remember Who is the root? Where is the root? What is the PIM F neighbor towards the root? BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 118

119 Troubleshooting Toolbox Checklist IGMP Membership Report Received by LHR WHO: is the root? WHERE is the root? WHAT is the PIM F neighbor towards the root? BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 119

120 NO Receivers Getting Mcast FHR Start at for this Group *,G w/ OIL? NO? Pick 1 LHR and start there S,G NO? Go to FHR LHR BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 120

121 Some Receivers Getting Mcast FHR LHR Start at LHR of Broken Receiver Pull? Who, Where, What BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 121

122 SSM Troubleshooting with Fish LHR FHR Start at FHR S,G w/ OIL? NO? Pick 1 LHR and start there YES? Check to see if traffic coming in from Source BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 122

123 Questions?

124 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card. Complete your session surveys through the Cisco Live mobile app or on Don t forget: Cisco Live sessions will be available for viewing on demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public

125 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions BRKIPM Cisco and/or its affiliates. All rights reserved. Cisco Public 125

126 Thank you

127