FabricPath Operation and Troubleshooting

Transcription

1

2 FabricPath Operation and Troubleshooting Carlo Schmidt, Customer Support Engineer BRKDCT-3313

3 Acronyms / Definitions Acronyms Definitions Acronyms Definitions ACL Access Control List FP FabricPath ASIC Application Specific Integrated Circuit FTAG Forwarding Tag ASID Anycast Switch Identifier LID Local Identifier BD Bridge Domain LTL Local Target Logic CE Classical Ethernet MIM MAC-in-MAC (common reference to FP header) DBUS / RBUS Data Bus / Result Bus PACL Port-based ACL DRAP Dynamic Resource Allocation Protocol RACL Router-based ACL DSID Destination Switch Identifier RPF Reverse Path Forwarding ELAM Embedded Logic Analyzer Module SoC Switch-On-Chip ES Emulated Switch SSID Source Switch Identifier FE Forwarding Engine VACL Vlan-based ACL FF Flood to Fabric VDC Virtual Device Context Reference Slide BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 3

4 Agenda FabricPath Overview Benefits, Restrictions, and Configuration Key Concepts Encapsulation, Trees, Topologies, STP Data Plane Forwarding, Load-Balancing, MAC Learning vpc+ Challenges and Operation Troubleshooting Verification steps, tools, and examples

5 FabricPath Benefits Existing Layer2 Single path between 2 points in L2 network Stability/Resilience at scale Disruptive convergence FabricPath Shortest path between switches + equal-cost load-balancing Core does not need to learn end host MAC addresses More resilient to loops No topology constraints, L3 anywhere Easy scaling / Non-disruptive merge BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Fabricpath Overview Unicast: Known Destination MAC Ingress FabricPath (Edge) Switch SSID comes from S10 s own switchid DSID comes from MAC address table for MAC B Intermediate switches forward based on DSID TTL decremented at every FP switch Egress FabricPath (Edge) Switch DSID 20 SSID 10 DMAC B DMAC B SMAC A DMAC B SMAC A SMAC A Payload Payload Payload MAC A MAC B CE FabricPath CE BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 FabricPath Overview Multidestination (broadcast, multicast, unicast flood) Ingress FP Switch selects Tree (FTAG) SSID comes from S10 s own switchid MAC B is unknown DSID = FloodSID Root switch for Tree 1 Root switch for Tree 2 FabricPath interface DMAC B SMAC A DMAC B SSID FTAG 1 DMAC B SMAC A Payload DMAC B SSID FTAG 1 DMAC B SMAC A Payload DMAC B SMAC A Payload CE interface Tree 1 Tree 2 Payload MAC A MAC B CE FabricPath CE BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 FabricPath support & configuration N7K with N7K-F1 linecard as of N7K with N7K-F2 linecard as of N7K + FEX as of (with N7K-F2) for CE ports F2E as of N7K with N7K-F3 linecard as of N5500 as of no L3 module required N FEX as of for CE ports N6K as of Enhanced L2 license required FabricPath Packaged as feature-set (plugin) N7K(config)# install feature-set fabricpath N7K(config)# feature-set fabricpath N7K(config)# interface Ethernet4/1 N7K(config-if)# switchport mode fabricpath... N7K(config)# vlan 3002 N7K(config-vlan)# mode fabricpath BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 FabricPath & CE Vlans Two types of vlans CE (Classic Ethernet, default) FabricPath (FP) FabricPath Classic Ethernet FP vlans cannot go on M1, M2 modules Only FP vlans will be carried over FP interfaces FP vlans can be mixed with CE vlans on edge interfaces Core = switchport mode fabricpath Edge = switchport mode access trunk N7K(config)# vlan 3002 N7K(config-vlan)# mode? ce Classical Ethernet VLAN mode fabricpath Fabricpath VLAN mode Port Type VLANs allowed to be configured N7K-M1, N7K-M2 FP, CE CE N7K-F1, N7K-F2, N7K-F3 Edge FP, CE FP, CE N7K-F1, N7K-F2, N7K-F3 Core FP, CE FP N5500, N6000 Edge FP, CE FP, CE N5500, N6000 Core FP, CE FP VLANs allowed to be brought up BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 Agenda FabricPath Overview Benefits, Restrictions, and Configuration Key Concepts Encapsulation, Trees, Topologies, STP Data Plane Forwarding, Load-Balancing, MAC Learning vpc+ Challenges and Operation Troubleshooting Verification steps, tools, and examples

11 Encapsulation Outer DA (48) Outer SA (48) FP Tag (32) DMAC SMAC 802.1Q Etype Payload CRC (new) Outer SA: 47 Endnode ID [ 5:0] U/L I/G Endnode ID [ 7:6] SwitchID ingress FP switch system ID SubswitchID is used in some cases of VPC+ LID is specific to the implementation N7K the LID is generally the port index of the ingress interface N5K/N6K LID most of the time will be 0 EndnodeID is not currently used 2 R S V D O O O 1 1 Switch ID 12 bits SubSwitch ID 8 bits Local ID 16 bits N7K# show fabricpath switch-id include SYS \* Legend: '*' - this system 0 SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED *2028 b414.89e3.a041 Primary Confirmed No No Outer DA: For known SA/DA is taken from MAC table for DMAC For broadcast and multicast is the same as DMAC For unknown unicast DA is 010f.ffc1.01c0 (flood to vlan) For known unicast DA, but unknown SA is 010f.ffc1.02c0 (flood to fabric) Example Ethernet II, Src: 02:00:64:01:FF:FF, Dst: 01:00:5e:00:00:02, Type: 0x8903 N7K# sh mac address-table address VLAN MAC Address Type age Ports/SWID.SSID.LID dynamic Switch_ID SubSwitch_ID LID BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 FabricPath Switch IDs, System IDs and DRAP Each FP switch is identified by unique number (ID), dynamically assigned or static Dynamic Resource Allocation Protocol (DRAP) is responsible for allocating switch IDs and resolving duplicate-id conflicts. Conflicts are resolved by renumbering switches with higher systemid (DRAP can only auto resolve non-static switch ID) N7K# show fabricpath switch-id FABRICPATH SWITCH-ID TABLE ========================================================================= SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED *3 c062.6bac.e343 Primary Confirmed Yes No f.ee02.ce3c Primary Confirmed Yes No f.ee04.5cfc Primary Confirmed Yes No When partitioned FP network is merged (or new switch joins the fabric) connecting interface is not enabled for data before all conflicts are resolved N7K(config-if-range)# no shut %FABRICPATH-2- FABRICPATH_LINK_BRINGUP_STALLED_STATIC: Link bringup stalled due to conflicts N7K# show fabricpath conflict all Port State Ethernet3/31 Suspended due to conflicts ============================================== Fabricpath Conflicts SYSTEM-ID SWITCH-ID STATIC c062.6bac.e343 3 Yes c062.6bac.e342 3 Yes = BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Network Merges / Conflict resolution Goal is to connect two networks with conflicting switch IDs without incurring packet loss 1) Allocate new switch-id as secondary tentative Wait allocate delay time 2) Make new switch-id as secondary - confirmed Wait transition delay time 3) Swap primary and secondary switch-ids Wait transition delay time 4) Delete old switch-id (now a secondary switch-id) More About Graceful Merge Graceful merge changes the switch-id of a switch to resolve switch-id collisions The switch-id to change is based on the system-id being higher value, or being dynamic For a time period the switch is identified by two switchids, packets for both are accepted but outgoing packets only carry the primary switch-id N7k# show fabricpath switch-id Legend: '*' - this system SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED *332 b414.89e3.a042 Primary Confirmed Yes No N7k# show fabricpath isis switch-id Legend: C - Confirmed, T - tentative, W - swap S - sticky, E - Emulated Switch '*' - this system System-ID Primary Secondary Reachable Bcast-Priority MT-0 b414.89e3.a042* 332 [C] 0[C] Yes 222 [S] N7k# show fabricpath timers Allocate Delay Timer : 10 Transition Delay Timer : 10 Link-up Delay Timer : 10 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 FabricPath Trees Known unicast traffic is load-balanced across equal-cost routes FabricPath uses two loop-free trees for unknown unicast, broadcast and multicast traffic Two trees are for load-balancing For each packet, tree is selected by ingress FP switch and choice is carried in the packet header Root of tree1 is the switch with highest Priority (highest sysid for tie) Root of tree2 is the switch with 2 nd highest Priority (highest sysid for tie) Tree is a least-cost-to-the-root graph, with lower sysid used as tie-breaker In case of Tree1 root failure both roots are reelected Up to 16 trees starting in 7.0 on Nexus 5000 and 6000 S2 SysID 10 FabricPath interface Tree 1 Tree 2 S1 SysID 50 R Lower SysID wins S4 SysID 30 S3 SysID 20 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Root Election / Tree construction Every switch advertises its system ID & Priority Once all nodes have spoken Broadcast Root is elected (Highest priority then Highest Mac address wins) Broadcast root system will Elect & Advertise Roots for additional multicast Trees (currently only 2 trees) Each node will independently run SPF with Tree Root and create 2 Trees Since Multicast roots are advertised by Broadcast Root system (Tree 1), in case of failure of the latter both Tree 1 and Tree 2 will re-converge S101# show fabricpath isis database detail Fabricpath IS-IS domain: default LSP database LSPID Seq Number Checksum Lifetime A/P/O/T S x000000E2 0x0FBB /0/0/1 Instance : 0x000000DD Area Address : 00 NLPID : 0xC0 Hostname : S1 Length : 2 Extended IS : S Metric : 40 Extended IS : S Metric : 40 Extended IS : S Metric : 40 Extended IS : S2.00 Metric : 40 Extended IS : S Metric : 40 Capability : Device Id: 1 Base Topology Base Topo Ftag : Graph 1: Root: S1 Primary: 1, Secondary: 0 Nickname 1 Graph 2: Root: S2 Primary: 2, Secondary: 0 Nickname 2 Base Topo Trees : Trees desired: 2 Trees computed: 2 Trees usable: 2 Base Topo Roots : Graph 1: Root Nickname: 1 Graph 2: Root Nickname: 2 Version : Version: 1 Flags: 0 Nickname : Priority: 0 Nickname: 1 BcastPriority: 255 Nickname Migration : Swid: 1 Sec. Swid: 0 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Encapsulation Outer DA (48) Outer SA (48) FP Tag (32) DMAC SMAC 802.1Q Etype Payload CRC (new) Ethertype 0x8903 FTAG TTL Ethertype for FabricPath packets is 0x bits 10 bits 6 bits TTL set to 32 and is decremented at every hop. Packet is discarded when TTL reaches 0. FTAG: (Forwarding TAG) Used for multidestination traffic; carries the ID of the tree chosen at the FabricPath ingress switch. DRAP is responsible to keep FTAGs unique/consistent. For known unicast, FTAG carries topology ID Nexus# show fabricpath isis topology summary Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary MT-0 Configured interfaces: Ethernet4/4 Root for Tree 1, FTAG 1 Number of trees: 2 Tree id: 1, ftag: 1, root system: 001b.54c2.4244, 4 Tree id: 2, ftag: 2, root system: 001b.54c2.4243, 3 Root for Tree 2, FTAG 2 Wireshark decodes FP encapsulation (tested on 1.8.3) : Edit Preferences Protocols CFP Enable Dissector BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Reverse Path Forwarding Check RPF: check where the source switch of the packet is and only accept packets from the interface we would have used if we were to send packet to that source At each FP hop RPF check is performed for multidestination traffic against source switchid + FTAG N7K# show l2 multicast trees root Accept packets from 1,4 Accept packets from 3 Accept packets from 4,1,2 May also use show fabricpath isis trees Packets with FTAG==2 from switch 30 will be accepted from interface e3/35 Packets with FTAG==1 from switch 30 will be accepted from interface e3/35 (ftag/2, topo/0, Switch-id 40), uptime: 1w0d, isis Outgoing interface list: (count: 1, '*' is the preferred interface) * Interface Ethernet3/39, [admin distance/115] uptime: 1d23h, isis (ftag/2, topo/0, Switch-id 30), uptime: 1w0d, isis Outgoing interface list: (count: 1, '*' is the preferred interface) * Interface Ethernet3/35, [admin distance/115] uptime: 02:56:04, isis (ftag/2, topo/0, Switch-id 100), uptime: 1w0d, isis Outgoing interface list: (count: 1, '*' is the preferred interface) * Interface Ethernet3/39, [admin distance/115] uptime: 1d23h, isis (ftag/1, topo/0, Switch-id 30), uptime: 02:56:06, isis Outgoing interface list: (count: 1, '*' is the preferred interface) * Interface Ethernet3/35, [admin distance/115] uptime: 02:56:06, isis BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Topologies + Vlans Flood/Multicast/Broadcast trees are per-vlan, made by pruning Topology Tree If vlan is not present on the switch, that switch will not be part of per-vlan tree This may lead to connectivity issues when not all transit switches in topology have all vlans similar to connectivity issues caused by liberal pruning vlans off trunks with MST Make sure each vlan exists in every transit switch in a topology Or, use mode Transit! Topology Tree VL10 VL30 VLAN 10 VLAN 20 VLAN 30 VL10 VL20 VL30 VL10 VL20 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Topologies Routing table & Trees (FTAGs) are per topology Default Topology allowed on all FP links Switch ID is shared across all topologies FP interface may belong to several topologies N7K: up to 8 topologies support starting in 6.2 FP links in Topology 0 and Topology 1 N5K/N6K: As of default + 1 extra topology is supported; main use is to permit separate L2 pods to use same local vlan set N7K# show fabricpath topology vlan Topo-Description Topo-ID Configured VLAN List , N7K# show fabricpath topology interface Interface Topo-Description Topo-ID Topo-IF-State port-channel1 0 0 Up Ethernet6/4 0 0 Up Ethernet6/5 0 0 Up port-channel1 1 1 Up Pod 1 Vlan Vlan Pod 2 Vlan Vlan fabricpath topology 1 member vlan ! interface Port-channel1 switchport mode fabricpath fabricpath topology 1 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 FabricPath Software Architecture & Hardware tables on the Supervisor Engine: FabricPath IS-IS routing protocol process that forms the core of the FabricPath control plane DRAP Dynamic Resource Allocation Protocol, ensures network-wide unique and consistent Switch IDs and FTAGs Resolves switch id conflicts U2RIB Unicast Layer 2 RIB, containing the best unicast Layer 2 routing information L2FM Layer 2 forwarding manager, controls MAC address table on the Linecards: U2FIB Unicast Layer 2 FIB, managing the hardware unicast routing table MTM MAC Table Manager, managing the hardware MAC address table DRAP FabricPath IS-IS U2RIB U2FIB Hardware Drivers Switch Table Other HW L2FM MTM Supervisor Engine MAC Table I/O Module BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Fabric Path Control Plane initialization flow S101# show processes cpu egrep "2rib drap fab l2fm PID" PID Runtime(ms) Invoked usecs 5Sec 1Min 5Min TTY Process % 0.00% 0.00% - l2fm % 0.00% 0.00% - m2rib % 0.00% 0.00% - u2rib % 0.00% 0.00% - isis_fabricpath % 0.00% 0.00% - drap Processes start (isis, u2rib, m2rib, drap) System ID obtained from backplane MAC S101# show fabricpath isis Fabricpath IS-IS domain : default System ID : 8478.ac0e.4743 IS-Type : L1 Fabric-Control SVI: Unknown... Process is up and running... Interfaces supported by Fabricpath IS-IS : port-channel1 Ethernet6/27 Ethernet6/ S101# show fabricpath switch-id Legend: '*' - this system SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED/ ANYCAST * ac0e.4743 Primary Confirmed Yes No... Switch ID is obtained from DRAP As FP interfaces links come up, hellos sent and adjacencies formed Switch ID conflicts (if any) resolved FP Interfaces allowed to forward data Unicast SPF is calculated Routes installed to U2RIB BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Fabric Path Control Plane initialization flow S101# show fabricpath isis interface Fabricpath IS-IS domain: default Interface: port-channel1 Status: protocol-up/link-up/admin-up LSP interval: 33 ms, MTU: 1500 P2P Adjs: 1, AdjsUp: 1, Priority 64 Hello Interval: 10, Multi: 3, Next IIH: 00:00:03 Level Adjs AdjsUp Metric CSNP Next CSNP Last LSP ID Inactive ffff.ffff.ffff.ff-ff Topologies enabled: Level Topology Metric MetricConfig Forwarding no UP no UP S101# show fabricpath isis adjacency Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database: System ID SNPA Level State Hold Time Interface S102 N/A 1 UP 00:00:25 port-channel1 S1 N/A 1 UP 00:00:28 Ethernet6/27 S2 N/A 1 UP 00:00:27 Ethernet6/ S101# show fabricpath isis spf-log Fabricpath IS-IS domain: default SPF information SPF log for Topology 0 Total number of SPF calculations: 55 Log entry (current/max): 20/20 Ago Level Reason Count Total 1d09h 1 New LSP S d09h 1 Updated LSP S BRKDCT-3313 Processes start (isis, u2rib, m2rib, drap) System ID obtained from backplane MAC Switch ID is obtained from DRAP As FP interfaces links come up, hellos sent and adjacencies formed Switch ID conflicts (if any) resolved FP Interfaces allowed to forward data Unicast SPF is calculated Routes installed to U2RIB 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Fabric Path Control Plane initialization flow S101# show fabricpath isis route Fabricpath IS-IS domain: default MT-0 Topology 0, Tree 0, Swid routing table 1, L1 via Ethernet6/27, metric 40 2, L1 via Ethernet6/28, metric , L1 via Ethernet6/27, metric 80 via Ethernet6/28, metric 80 How to read... To reach switch 200 in topology 1 send packets to either Eth6/27 or Eth6/28 S101# show fabricpath route FabricPath Unicast Route Table 'a/b/c' denotes ftag/switch-id/subswitch-id '[x/y]' denotes [admin distance/metric] ftag 0 is local ftag FabricPath Unicast Route Table for Topology-Default... 1/102/0, number of next-hops: 1 via Po1, [115/40], 1 day/s 10:01:12, isis_fabricpath-default 1/200/0, number of next-hops: 2 via Eth6/27, [115/80], 1 day/s 10:02:32, isis_fabricpath-default via Eth6/28, [115/80], 0 day/s 10:20:17, isis_fabricpath-default Processes start (isis, u2rib, m2rib, drap) System ID obtained from backplane MAC Switch ID is obtained from DRAP As FP interfaces links come up, hellos sent and adjacencies formed Switch ID conflicts (if any) resolved FP Interfaces allowed to forward data Unicast SPF is calculated Routes installed to U2RIB BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 FabricPath IP Multicast Control plane: IGMP snooping operates as usual in FabricPath edge switches FabricPath IS-IS learns multicast group membership from IGMP snooping on edge switch FabricPath edge switch announces group interest by using GM-LSPs, creating pruned trees for each group on each multidestination tree Data plane: Hardware selects which multidestination tree to use for each flow based on hash function Once tree is selected, traffic constrained to pruned tree (FTAG) for that IP multicast group, based on MAC table lookup BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Key FabricPath Multicast Processes on the Supervisor Engine: FabricPath IS-IS routing protocol that forms the core of the FabricPath control plane DRAP Dynamic Resource Allocation Protocol, extension to FabricPath IS- IS that ensures network-wide unique and consistent Switch IDs and FTAGs IGMP Provides IGMP snooping support for building multicast forwarding database M2RIB Multicast Layer 2 RIB, contains the multicast Layer 2 routing information L2FM Layer 2 forwarding manager, controls the MAC address table MFDM Multicast forwarding distribution manager, connects platformindependent control-plane processes and platform-specific processes on I/O modules on the Linecards: M2FIB Multicast Layer 2 FIB, manages the hardware multicast routing table MTM MAC table manager, manages the hardware MAC address table DRAP FabricPath IS-IS IGMP M2RIB MFDM M2FIB Hardware Drivers Switch Table Other HW L2FM MTM MAC Table Supervisor Engine I/O Module BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 FabricPath Multicast Control Plane IGMP/IGMP snooping tracks connected hosts/routers interest in receiving multicast S10 S1 S2 S30 ISIS distributes information from igmp snooping to other FP nodes using GM-LSPs. Intermediate nodes flood GM-LSPs Receiver S20 Receiver A pruned subtree is created for each group (+flood, OMF) per vlan per FTAG Vlan FTAG MAC Switches Interfaces e S10,S30 E1/10,E1/ e S10,S30 E1/2 Source Vlan FTAG MAC Switches Interfaces e S10,S30 E1/ e S10,S30 E1/10,E1/30 Root Tree1 S1 E1/2 S2 S1 E1/1 S2 Root Tree2 S10 E1/10 E1/30 S30 S10 E1/10 E1/30 S30 MAC A S20 S20 E1/1 MAC B MAC A E1/2 MAC B Vlan FTAG MAC Switches Interfaces e S10,S30 E1/1 Vlan FTAG MAC Switches Interfaces e S10,S30 E1/2 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 STP & FabricPath No STP inside FP network BPDUs do not traverse FP network (dropped at FP edge, with the exception of TCNs, see next slide) FP network pretends to be 1 switch from STP point of view: all FP edge switches send BPDUs with the same Bridge ID c84c.75fa.60xx (xx is domain ID in hex, default 00) Before FP ports are up, switch will use its own Bridge ID (like STP without FP would do) Ports inside FP cannot be blocked, FP edge switches will always want to have STP designated role, if superior BPDU is received such port will be blocked as L2GW inconsistent FabricPath N7K# show spanning-tree interface e3/1 detail Port 385 (Ethernet3/1) of VLAN2000 is broken (L2 Gateway Backbone Port Inconsistent) Designated root has priority 34768, address c84c.75fa.6000 N7K(config)# spanning-tree vlan 2000 priority :27:28 %STP-2-L2GW_BACKBONE_UNBLOCK: L2 Gateway Backbone port inconsistency cleared unblocking port Ethernet3/1 on VLAN2000. BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 T C N STP, FabricPath & TCNs When CE STP domains are connected to multiple FP switches STP TCN handling might be needed to maintain accuracy of MAC address tables inside CE Example if link CE1-CE2 goes down, link CE2-CE3 will become forwarding. Now to reach MAC B, switches inside FP need to send traffic to S5 instead of S4 To achieve this, FP switches when receiving a TCN from CE will propagate it to all FP switches in the network (via ISIS) Each FP switch will flush all remote MAC addresses learned from switches in the same STP domain as domain originating the TCN In addition, if FP switch is also part of the same STP domain, it will propagate TCN to the CE domain TCNs are not propagated to CE in domain 0 (default domain) N7K# conf t N7K(config)# spanning-tree domain? <1-1023> Domain Identifier N7K# sh spanning-tree summary Switch is in rapid-pvst mode L2 Gateway Domain ID: Flush MACs learned from S4,S5 MAC A S3 S1 S4 CE1 T C N T C N T C N STP Domain 1 FabricPath MAC B S2 T CS5 STP Domain 2 N X CE2 CE3 Flush MACs learned on CE T C N BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Control Plane Protection Both N7K, N6K, and N5K recognize and protect FP ISIS traffic at COPP level COPP needs to be updated when deploying FabricPath; standard profiles are FP-aware as of 5.2(1) N7K# show policy-map interface control-plane Control Plane 7K service-policy input: copp-policy-strict class-map copp-class-critical (match-any) match access-group name copp-acl-mac-fabricpath-isis set cos 7 police cir kbps, bc 250 ms module 1 : conformed bytes; action: transmit violated 0 bytes; action: drop N5K# show policy-map interface control-plane class copp-system-class-isis Control Plane service-policy input: copp-system-policy-default class-map copp-system-class-isis (match-any) match protocol isis_dce police cir 1024 kbps, bc bytes 5K conformed bytes; action: transmit violated 0 bytes; 6K In case of complex CE-side STP topologies (with blocking ports), usual STP safeguards are recommended (Bridge Assurance & Dispute / UDLD) On N7K-F1 cards: rate-limiters allow up to 4500 PPS worth of control plane FabricPath packets Note: These 4500 PPS include also transit packets BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 Agenda FabricPath Overview Benefits, Restrictions, and Configuration Key Concepts Encapsulation, Trees, Topologies, STP Data Plane Forwarding, Load-Balancing, MAC Learning vpc+ Challenges and Operation Troubleshooting Verification steps, tools, and examples

31 FabricPath: Forwarding Tables FabricPath uses 3 tables to forward frames MAC address table VLAN, MAC Address, Port (local or remote), FTAG (for non-unicast) Switch-ID table remote switch-id, local next-hop interfaces (up to 16) Multidestination tree table Per Tree: remote switch-id, local next-hop/rpf interface Tree#1 (broadcast, unknown unicast, IP multicast) Tree#2 (IP multicast) DRAP FabricPath IS-IS U2RIB U2FIB Hardware Drivers Switch Table Other HW L2FM MTM Supervisor Engine MAC Table I/O Module BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 Forwarding: unicast CE FP unicast This is meant to illustrate key decisions in forwarding, some details are abstracted away DA Known N Unknown unicast ODA = MC1 (Flood2BD) Y SA Known Y ODA = L2_lookup (DA) N Unknown source Flood to update MACs ODA = MC2 (FF) DA = Destination Address SA = Source Address ODA = Outer Destination Address OSA = Outer Source Address MC1 = 010F.FFC1.01C0 MC2 = 010F.FFC1.02C0 FTAG for unicast is topology ID Ftag = Vlan2Ftag(Vlan) Choose FTAG Ftag = F(Vlan,SA/DA, ) TTL = 32 OSA.SW/SubSW = local OSA.LID=LID(ingress_port) Forward BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 Forwarding: broadcast/multicast CE FP BC MC ODA = DA Ftag = Hash(Vlan,SA/DA, ) TTL = 32 Broadcasts are flooded along FTAG1 * Exception in vpc+ OSA.SW/SubSW = local OSA.LID=LID(ingress_port) Forward Frame is flooded on CE side as well (based on DA) Each egress port decides whether to encapsulate the frame in MIM depending on port type (FP,CE) BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Forwarding: FP->FP or FP->CE MIM packet Decrement(TTL) RPF is checked against OSA.SwID + FTAG Y TTL<1 N Destination = Sw_Table(FTAG, ODA.SwID) N ODA is unicast Y ODA.SwID is local N Pass RPF Fail Destination = L2_Table(Vlan, FTAG, ODA) Multicast lookups are done using VLAN, FTAG, and ODA (each multicast mac appears twice) SubSwitchID lookups are omitted here Y Dest = LID or Dest = L2_table(DA,VLAN) Forward Remember about special LIDs (Sup, Flood, ) FF frames are forwarded out of CE ports only when DA is locally learned Drop BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 Load-balancing N7K: Unicast and Multicast load-balancing are separate N5K/N6K: Unified load-balancing mechanism for unicast and multicast N7K# show fabricpath load-balance ECMP load-balancing configuration: L3/L4 Preference: Mixed Hash Control: Symmetric Rotate amount: 6 bytes Use VLAN: TRUE Ftag load-balancing configuration: Hash Control: Symmetric Rotate amount: 6 bytes Use VLAN: TRUE Symmetric: idea is to make a b and b a flows take same path by sorting addresses, before feeding them to hash Rotate: polarization avoidance; hash result is rotated by specified number of bytes. Number is derived from unique system MAC N7K# show fabricpath load-balance unicast forwarding ftag 1 switchid 30 flow l2 src-mac 001c.57ad.ecc3 dst-mac 547f.ee02.ce3c ether-type 0x800 vlan 2000 module 3 128b Hash Key generated : 1ffb80b38f eb7b30d5 This flow selects interface Eth3/25 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 Reducing impact of forwarding loops Transient loops might occur during convergence (as with L3 routing) To contain impact of these loops FabricPath uses TTL. Starting in 6.2(2), can set the initial TTL via fabricpath [multicast unicast] ttl For Multidestination Trees Reverse Path Forwarding check performed on source switch ID Nexus5k# show platform fwm info asic-errors 0 DROP_TTL_EXPIRED: res0 = 23 res1 = 0 [10] Nexus7K-F2# show hardware internal errors module 4 inc ign ttl 47 Ingress redirect due to dtag_ttl check BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 MAC Address Learning Learning MAC addresses is not required in FabricPath Core as switching is based on Switch ID FP Edge switches learn local MAC addresses (behind edge ports) conventionally FP Edge devices learn remote addresses (behind Core-facing ports) using conversational learning For packets arriving from FP, source MAC (not outer SA!) is learned when destination MAC of the frame is already known on any Edge port of this switch No learning from broadcasts (though existing entries will be updated) Normal Learning from multicasts (example: HSRP address) Conversational learning is disabled on L3 edge switches (when SVI is up on FP VLAN) This does not apply to a case where F-series is connected to M-series in different VDC by external cable When M and F are in the same VDC, special handling is needed to forward packets from M FP core this is orchestrated by MCM (mixed chassis manager) BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 Conversational MAC Address Learning A S1 S2 S3 MAC Port MAC Port MAC Port B A sends an ARP for B (broadcast) S1 S2 S3 A MAC Port MAC Port MAC Port A 1 B sends ARP reply (unicast) to A S1 S2 S3 A MAC Port MAC Port MAC Port A 1 B 1 B S3.0.1 A sends unicast packet to B A S1 S2 S3 MAC Port MAC Port MAC Port A 1 B 1 B S3.0.1 A S1.0.1 B B B BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 FabricPath Scale Nexus5500 Nexus6000 N7K-F1 N7K-F2 N7K-F3 N7K-M series 32K MACs 128K MACs* 16K MACs per SoC 16K MACs per SoC 64K MACs per SoC 128K MACs Potential bottleneck if F1/F2 used in L3 Spine L3 Spine L3 Spine L2 Spine Leaf Leaf VLAN 100 VLAN 200 VLAN 100 VLAN 200 VLAN 100 VLAN 200 VLAN 100 VLAN 200 Leaf Layer Optimized conversational learning Spine No MAC learning (forwarding based on SWID) Leaf Layer Optimized conversational learning Spine Learns all MAC addresses in order to route between VLANs BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 FabricPath Proxy L2 Learn Goal: Increase MAC table size in FabricPath for F1/F2E modules Solution: Offload MAC learning to M-series module at L2/L3 boundary Prerequisites: 6.2(2) on N7K (Spine and Leaf), M1/M2 + F2E or M1/M2 + F1 Configuration L3 M1/M2 Learn All Remote MACs! From default VDC (Prevents F2E/F1 from learning on multicast frames) no hardware fabricpath mac-learning module <x> [port-group <y>] Spine L2 SoC No MAC Learning! From fabricpath VDC (prevents F2E/F1 from learning remote MACs) no mac address-table fabricpath remote-learning Leaf VLAN 100 VLAN 200 VLAN 100 VLAN 200! If you are using F2 for Leaf core ports to prevent learning from broadcast/multicast no hardware fabricpath mac-learning module <x> [port-group <y>] BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 FabricPath MAC Learning Changes: Why? S101 FP SoC CE SoC VLAN MAC Index S1 L3 L2 200 A gpc1 M1/M2 FP SoC FP SoC CE SoC VLAN 100 VLAN 200 X, Y, Z A, B, C M sends frame to gpc1 GPC gpc1 S201 SWID S201 F translates frame to SWID 201, LID FFFFMAC miss, causes flood to local CE ports M-Series MAC tables contain VLAN, MAC, and port index (no concept of SWID, SSWID, LID in M-Series MAC table) For FP MACs, the destination SWID is mapped to an internal gateway port-channel (GPC) index which is programmed in the M-series MAC table FP SoC will translate GPC to SWID before sending out FP port. Challenge: No way for FP SoC to determine LID for packet from M-Series module if MAC is not present in local MAC table. Therefore, packet from M-Series sent out FP with flood LID. If FP SoC on destination switch has not learned MAC, then packet will be flooded out local CE ports Solution: Sync MACs on CE SoC to FP SoC. BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 FabricPath MAC Learning Changes To support L2 proxy learning, MACs learned on CE ports will be synced to all SoCs CE SoC A, B, C A, B, C CE SoC No MACs Learned FP SoC FP SoC Learn all MACs on CE ports. Learn remote MACs via conversational learning Learns MAC A,B,C Sync local CE MACs to FP SoC Learns MAC X,Y,Z FP SoC FP SoC Learn all MACs on CE ports. Learn remote MACs via conversational learning CE SoC 6.1(2) for F2/F2E 6.2(2) in F1 CE SoC X, Y, Z X, Y, Z BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Agenda FabricPath Overview Benefits, Restrictions, and Configuration Key Concepts Encapsulation, Trees, Topologies, STP Data Plane Forwarding, Load-Balancing, MAC Learning vpc+ Challenges and Operation Troubleshooting Verification steps, tools, and examples

44 VPC+: Why, What and How (1) Goal: provide redundant, active-active L2 links to separate FP switches with active-active HSRP Challenge 1: depending on the path the packet A B takes, switch S3 will learn MAC A behind S1 or S2 (or MAC will be moving) Solution: introduce Emulated Switch S100 to represent devices behind VPCs: MAC A will appear behind S100 in S3 MAC address table. HSRP MAC is advertised with emulated switch as a source taking advantage of VPC+ multipathing S3# show mac address-table address a VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID a dynamic 30 F F S3# show fabricpath route switchid 100 1/100/0, number of next-hops: 2 via e1/1, [115/20], 1 day/s 05:56:40, isis_fabricpath-default via e1/2, [115/20], 1 day/s 05:56:38, isis_fabricpath-default S3# show fabricpath switch-id SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED Primary Confirmed Yes No Primary Confirmed Yes No * Primary Confirmed Yes No Primary Confirmed No Yes S1 MAC B S3 Fabric Path S100 MAC A S2 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 VPC VPC+ Fabric Path To enable VPC+ an Emulated Switch ID must be configured in VPC domain on both peers (must be the same on both peers and globally unique). ES represents ALL VPC+ channels of the domain Peer-link and VPC+ ports must be fabric-path capable Peer-link is FP interface (no STP, only FP vlans are carried, VPC check is no more ). VPC+ channels are CE VPC+ domain must be the root for CE STP, otherwise VPC+ channels will be blocked as L2GW inconsistent FP switches use same STP bridge ID but peer-switch is still recommended S1 S100 S2 S1# show vpc vpc domain id : 2 vpc+ switch id : 100 Peer status : peer adjacency formed ok vpc keep-alive status : peer is alive vpc fabricpath status... vpc role : primary Number of vpcs configured : 0... Fabricpath load balancing : Disabled Port Channel Limit : limit to 244 : peer is reachable through fabricpath vpc domain 2 fabricpath switch-id 100 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 HSRP (and VRRP) in VPC+ HSRP when enabled on VPC+ peers uses Emulated Switch ID as a source switch and thus benefits from VPC+ multipathing Control-plane-wise one peer will be active and other will be standby, but data-plane-wise both peers will be forwarding traffic (same as in VPC) FabricPath devices will have ECMP route to Emulated Switch S3 S3# show mac address-table vlan 100 address c9f.f064 VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID c9f.f064 dynamic 0 F F Fabric Path s3# show fabricpath route switchid 100 1/100/0, number of next-hops: 2 via e1/1, [115/20], 1 day/s 05:56:40, isis_fabricpath-default via e1/2, [115/20], 1 day/s 05:56:38, isis_fabricpath-default S1 S2 CE devices will have HSRP VMAC pointing to a port-channel CE1# show mac address-table vlan 100 address c9f.f064 VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID * c9f.f064 dynamic 0 F F Po1 If only HSRP active-active is required VPC+ channels are optional CE1 S100 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 VPC+: Why, What and How (2) MAC B Challenge 2: flooded packets from A (with OSA of S100) might come to S3 from S1 or from S2, but RPF can only be 1 interface S3 Solution: S1 and S2 advertise to S3 (via ISIS TLV) an affinity to single FTAG each, S3 will program RPF according to affinity. Multidestination traffic coming from VPC+ will be set to use FTAG 1 for VPC leg on S1 and FTAG 2 for VPC leg on S2 S3# show fabricpath route switchid 100 FabricPath Unicast Route Table 1/100/0, number of next-hops: 2 via Eth1/1, [115/40], 11 day/s 00:59:35, isis_fabricpath-default via Eth1/2, [115/40], 11 day/s 01:03:27, isis_fabricpath-default S3# show fabricpath isis database detail i Affinity Host Numg Hostname : S1 Length : 2 Affinity : Nickname: 100 Numgraphs: 1 Graph-id: 1 Hostname : S2 Length : 2 Affinity : Nickname: 100 Numgraphs: 1 Graph-id: 2 S3# show l2 multicast trees (ftag/2, topo/0, Switch-id 100), uptime: 1d01h, isis Outgoing interface list: (count: 1, '*' is the preferred interface) * Interface Ethernet1/2, [admin distance/115] uptime: 1d01h, isis (ftag/1, topo/0, Switch-id 100), uptime: 6d00h, isis Outgoing interface list: (count: 1, '*' is the preferred interface) * Interface Ethernet1/1, [admin distance/115] uptime: 6d00h, isis 47 BRKDCT-3313 RPF FTAG1,S100 Affinity FTAG1 S1 Use FTAG1 1/1 1/2 MAC A 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public RPF FTAG2,S100 S100 Affinity FTAG2 S2 Use FTAG2 47

48 VPC+: Why, What and How (3) Challenge 3: multidestination packets from FP to CE need to be loadbalanced too Solution: S1 and S2 will each be designated forwarder for FTAG of their affinity: traffic for FTAG of affinity will be forwarded out of VPC and other FTAG traffic will be forwarded by peer S1# show vpc vpc domain id : 100 vpc+ switch id : vpc Peer-link status Po1 up , vpc status id Port Status Consistency Reason Active vlans vpc+ Attrib Po101 up success success 10 DF: Yes RPF FTAG1,S100 Affinity FTAG1 S1 DF: FTAG1 MAC B S3 1/1 1/2 Po101 RPF FTAG2,S100 S100 Affinity FTAG2 S2 DF: FTAG2 vpc status id Port Status Consistency Reason Active vlans vpc+ Attrib Po101 up success success 10 DF: Partial vpc domain 100 fabricpath multicast load-balance MAC A BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 VPC+: Prevention of Duplicate Packets How is packet received from VPC+ and flooded on S1 prevented from being flooded on S2 to same VPC+ again? N7K-F1 linecards: Each VPC+ will have its own sub-switch ID. Mac addresses will be learned behind <es_id>.<subsw_id>.<lid>, for example (emulated switch 100, sub-switch 11, LID 65535). S2 will recognize ES + SubSwitch tuple as its own port and will not flood the frame back to VPC Fabric Path N7K-F2, N7K-F3 linecards & N5K, N6K: By default same as above, as below with fabricpath multicast load-balance Each VPC+ peer will be forwarding only for 1 FTAG and traffic coming from other peer will have different FTAG. For example (previous slide) flooded packet coming from S1 will have FTAG1, but S2 will only flood FTAG2 packets out of the VPC S1 X S2 Required for FEX FP with N7K-F2 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 VPC Failover VPC+ member link goes down Traffic diverted over Peer-Link Peer-Link goes down (but Peer-Keepalive up) Primary: No action Secondary: Bring down VPC+ channels Stop advertising reachability to Emulated Switch S3# show fabricpath route switchid 100 1/100/0, number of next-hops: 1 via e1/1, [115/20], 1 day/s 07:14:24, isis_fabricpath-default Dual active is much less likely than with normal VPC: if Peer-Link and Peer-Keepalive go down, but peer is reachable via FP secondary will not become primary S1# show vpc vpc domain id : 2 vpc+ switch id : 100 Peer status : peer adjacency formed ok vpc keep-alive status : peer is alive vpc fabricpath status : peer is reachable through fabricpath S1 Fabric Path S3 S100 S2 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 Anycast HSRP Goal: provide N-gateway solution to increase redundancy and bandwidth Alternatives: 1. vpc/vpc+ provides 2 active gateways. Failure of a single gateway reduces available inter-vlan traffic by half 2. GLBP allows more than 2 active gateways. Drawbacks: No ECMP load-balancing since a single virtual MAC is assigned to a single SwitchID Non-deterministic distribution of virtual MAC addresses (hard to troubleshoot) Solution: Anycast HSRP L3 L2 Active Standby Listen Listen Fabric Path All 4 devices actively routing traffic for the HSRP virtual MAC BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 Anycast HSRP The HSRP virtual MAC is bond to an Anycast SwitchID (ASID) ASID uses similar concept to vpc+ ES, where each Anycast gateway advertises the ASID via new Anycast HSRP Sub-TLV Each Anycast gateway will actively route traffic for the HSRP virtual MAC L3 L2 ASID S1 S2 S3 S4 Configure HSRP under the interface - HSRP version2 required feature interface-vlan feature hsrp interface Vlan100 ip address /24 hsrp version 2 hsrp 100 ip interface Vlan101 ip address /24 hsrp version 2 hsrp 101 ip Code Requirement N7K 6.2(6) N5K/N6K 6.0(2)N2(1) (SubTLV only) 7.0(0)N1(1) 4 Equal Cost Routes to ASID hsrp anycast 1 ipv4 switch-id 1000 vlan no shutdown Configured the ASID for this anycast bundle and associate vlans BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Anycast HSRP S202# show fabricpath isis database detail i "LSPID Nickname: 1000" LSPID Seq Number Checksum Lifetime A/P/O/T S x x815E 762 0/0/0/1 Nickname: 1000 Numgraphs: 2 Graph-id: 1, 2 S x xC /0/0/1 Nickname: 1000 Numgraphs: 2 Graph-id: 1, 2... S202# show fabricpath route switchid /1000/0, number of next-hops: 4 via Eth1/6, [115/40], 0 day/s 03:00:18, isis_fabricpath-default via Eth1/7, [115/40], 0 day/s 03:02:55, isis_fabricpath-default via Eth1/8, [115/40], 0 day/s 03:01:08, isis_fabricpath-default via Eth1/9, [115/40], 0 day/s 03:03:45, isis_fabricpath-default S202# show mac address-table dynamic Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vpc Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID * c9f.f065 dynamic 10 F F * c9f.f064 dynamic 10 F F Each switch sends ISIS TLVs advertising ASID ECMP routes built toward ASID to increase redundancy and bandwidth HSRP Active Hellos are sent out with a OSA of the ASID and SA of the virtual MAC BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 Agenda FabricPath Overview Benefits, Restrictions, and Configuration Key Concepts Encapsulation, Trees, Topologies, STP Data Plane Forwarding, Load-Balancing, MAC Learning vpc+ Challenges and Operation Troubleshooting Verification steps, tools, and examples

55 FabricPath: Configuration S1 S2 FP Vlans install feature-set fabricpath feature-set fabricpath S101 S102 S201 S202 vlan mode fabricpath Best practice to manually fabricpath switch-id 101 configure switch-id vpc domain 100 fabricpath switch-id 100 fabricpath multicast load-balance! Fabricpath core ports interface Ethernet6/4-5 switchport mode fabricpath! Peer-link interface port-channel1 switchport mode fabricpath! vpcs are CE ports (mode access or mode trunk) interface port-channel20 switchport switchport mode trunk vpc 20 A ES S100 B C ES S200 D! S1 fabricpath domain default root-priority 255! S2 fabricpath domain default root-priority 254 Configure roots for FTAG 1 and 2 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 FabricPath: Health Check S1 S2 FP Vlans S101# sh sys internal plugin info global begin l2mp head lines 5 Feature-set id: 2, name: l2mp vdc: 1 state: PLUGIN_ENABLED_STATE vdc: 2 state: PLUGIN_ENABLED_STATE vdc: 3 state: PLUGIN_ENABLED_STATE S101 S102 S201 S202 FabricPath plugin in good state Services running for URIB, MRIB, DRAP, ISIS CPU levels are reasonable Memory below limits A ES S100 B C ES S200 D S101# show system internal sysmgr service all i 2rib drap fabric PID Name UUID PID SAP state Start count Tag Plugin ID isis_fabricpath 0x s N/A 1 drap 0x E s N/A 1 m2rib 0x s N/A 1 u2rib 0x s N/A 1 S101# show processes cpu i 2rib drap fabric PID PID Runtime(ms) Invoked usecs 5Sec 1Min 5Min TTY Process % 0.00% 0.00% - m2rib % 0.00% 0.00% - u2rib % 0.00% 0.00% - isis_fabricpath % 0.00% 0.00% - drap S101# show processes memory i 2rib drap fabric PID PID MemAlloc MemLimit MemUsed StackBase/Ptr Process ffd8cb40/ffffffff m2rib ffbc5b80/ffffffff u2rib ff8eed50/ffffffff isis_fabricpath ffa58950/ffffffff drap BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 FabricPath: Health Check S1 S2 FP Vlans S101# show fabricpath isis System ID : 8478.ac0e.4743 IS-Type : L1 Fabric-Control SVI: Unknown Process is up and running Interfaces supported by Fabricpath IS-IS : port-channel1 Ethernet6/27 Ethernet6/28 S101 S102 S201 S202 S101# show fabricpath topology vlan active Topo-Description Topo-ID Active VLAN List A ES S100 B C ES S200 D ISIS is running system ID is accurate Interface list matches configuration Active Vlans match configuration Interfaces in Up/Ready state Adjacencies established Adjacencies stable S101# show fabricpath isis interface brief Fabricpath IS-IS domain: default Interface Type Idx State Circuit MTU Metric Priority Adjs/AdjsUp port-channel1 P2P 3 Up/Ready 0x01/L /1 Ethernet6/27 P2P 1 Up/Ready 0x01/L /1 Ethernet6/28 P2P 2 Up/Ready 0x01/L /1 S101# show fabricpath isis adjacency detail Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database: System ID SNPA Level State Hold Time Interface S102 N/A 1 UP 00:00:25 port-channel1 Up/Down transitions: 1, Last transition: 3w5d ago Circuit Type: L1 Topo-id: 0, Forwarding-State: UP BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 FabricPath: Health Check S1 S2 FP Vlans S101# show fabricpath isis traffic port-channel 1 Fabricpath IS-IS domain: default Fabricpath IS-IS Traffic for port-channel1: PDU Received Sent RcvAuthErr OtherRcvErr ReTransmit P2P-IIH n/a CSNP n/a PSNP n/a LSP S101 S102 S201 S202 S101# show fabricpath switch-id FABRICPATH SWITCH-ID TABLE Legend: '*' - this system '[E]' - local Emulated Switch-id '[A]' - local Anycast Switch-id Total Switch-ids: 10 ========================================================================= SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED/ ANYCAST ac0e.4742 Primary Confirmed Yes No ac5b.2b42 Primary Confirmed Yes No [E] ac0e.4743 Primary Confirmed No Yes ac5b.2b43 Primary Confirmed No Yes * ac0e.4743 Primary Confirmed Yes No ac5b.2b43 Primary Confirmed Yes No f.eed6.70fc Primary Confirmed No Yes f.eedb.7e7c Primary Confirmed No Yes f.eed6.70fc Primary Confirmed Yes No f.eedb.7e7c Primary Confirmed Yes No A ES S100 B C ES S200 D No growing errors on interfaces All switches and ES are seen and in confirmed state BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 FabricPath: Unicast Example (MAC) S101# show mac-address-table address-table vlan 100 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vpc Peer-Link, (T) - True, (F) - False VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID * a dynamic 0 F F Po d dynamic 0 F F S101 vpc30 S1 S102 S2 S201 FP Vlans A ES S100 B C ES S200 D S202 vpc40 S101# show hardware mac address-table 6 vlan 100 FE Valid PI BD MAC Index Stat SW... SWID SSWID LID ic a 0x x089 0x064 0x00b 0x d 0x x009 0x0c8 0x000 0x00000 S101# show system internal pixm info ltl 0x408 PC_TYPE PORT LTL RES_ID LTL_FLAG CB_FLAG MEMB_CNT Normal Po30 0x0408 0x d 0x x K 7K MACs are present in software MAC table Use Platform Dependent commands to check hardware MAC table On S101, MAC D matches software remote address ( ) MAC A has local SWID/SSWID with LID 0x408 LID 0x408 maps to local Po30 Hex SWID/SSWID 0x64 0x0b = xc8 0x00 = BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 FabricPath: Unicast Example (MAC) S1 S2 FP Vlans S202# show mac address-table vlan 100 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vpc Peer-Link VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID * a dynamic 0 F F * d dynamic 0 F F Po40 S101 vpc30 S102 S201 A ES S100 B C ES S200 D S202 vpc40 S202# show platform fwm info hw-stm i HW VLAN _ a 000d HW STM Contents dleft loc - bucket_type:line:bucket_number misc - learn_type:ecc:valid:fcf cdce format - ig:ul:switch_id:subswitch_id:end_node_id:pbp_idx:local_id VLAN MAC Address Port loc misc cdce d Po40 1:1111:0 1:0:1:0 2.0.c (e:0) a l2mp-nh 1:2918:0 1:0:1: b.ff.ff (e:0) S202# show platform fwm info lif port-channel 40 i local_id Po40 pd: local_id 21 endnode_id 0 endnode_id_alloced 1 vif_id 0 LID 21 maps to local Po40 5K 5K MACs are present in software MAC table Use Platform Dependent commands to check hardware MAC table On S202, MAC A matches software remote address ( ) MAC A has local SWID/SSWID with LID 0x15 (0x15 = 21) BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 FabricPath: What command comes from where show fabricpath switch show fabricpath conflict all link switch transitions show fabricpath isis switch show fabricpath isis interface show fabricpath isis adjacency show fabricpath isis database show fabricpath isis route DRAP FabricPath IS-IS U2RIB L2FM Supervisor Engine show fabricpath route show mac address-table slot <> show fabricpath unicast routes vdc slot <> show hardware internal forwarding inst <> table <> show platform fwm info l2mp route ftag <> switch <> hw 7K 5K 6K U2FIB Hardware Drivers Switch Table Other HW MTM MAC Table I/O Module slot <> show hardware mac address-table 7K show platform fwm info hw-stm 5K 6K BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 FabricPath: Unicast Example (SWID) S1 S2 FP Vlans S101# show fabricpath isis database detail Fabricpath IS-IS domain: default LSP database LSPID Seq Number Checksum Lifetime A/P/O/T S x xF8A /0/0/1 Hostname : S201 Length : 4 Capability : Device Id: 201 Base Topology Affinity : Nickname: 200 Numgraphs: 1 Graph-id: 1 Nickname : Priority: 0 Nickname: 201 BcastPriority: 64 Priority: 0 Nickname: 200 BcastPriority: 0 S x x5F3B 884 0/0/0/1 Hostname : S202 Length : 4 Capability : Device Id: 202 Base Topology Affinity : Nickname: 200 Numgraphs: 1 Graph-id: 2 Nickname : Priority: 0 Nickname: 202 BcastPriority: 64 Priority: 0 Nickname: 200 BcastPriority: 0 S101# show fabricpath isis route Fabricpath IS-IS domain: default MT-0 Topology 0, Tree 0, Swid routing table , L1 via Ethernet6/27, metric 80 via Ethernet6/28, metric 80 S101 vpc30 S102 S201 A ES S100 B C ES S200 D Route for destination SWID present in ISIS table and U2RIB S101# show fabricpath route switchid 200 FabricPath Unicast Route Table 'a/b/c' denotes ftag/switch-id/subswitch-id '[x/y]' denotes [admin distance/metric]... 1/200/0, number of next-hops: 2 via Eth6/27, [115/80], 0 day/s 00:21:58, isis_fabricpath-default via Eth6/28, [115/80], 0 day/s 00:21:58, isis_fabricpath-default S202 vpc40 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 FabricPath: Unicast Example (SWID) S1 S2 FP Vlans module-6# show fabricpath unicast routes vdc 3 ftag 1 switchid 200 Route in VDC FTAG SwitchID SubSwitchID Loc/Rem RPF RPF Intf Num Paths Merge V Remote Yes Eth6/ PD Information for ECMP: Common Info AMM key : 0x Next Hop Interface LID Eth6/ a 1 Eth6/ b Two equal costs routes via Eth6/27 and Eth6/28. RPF interface Eth6/27 S202# show platform fwm info l2mp route ftag 1 swid l2mp_route[0x99f23ac] route_type: 10 (0xa) merge_version: 1 (0x1) iic interface: Eth1/7 (0x1a006000) ftag: 1 (0x1) switchid: 100 (0x64)-> l2mp_nexthop[0x8944dc4] num_paths: 2 nh[1]: Eth1/7 (0x1a006000) nh[2]: Eth1/8 (0x1a007000) 5K 7K S101 vpc30 S102 S201 A ES S100 B C ES S200 D Two equal costs routes via Eth1/7 and Eth1/8. RPF interface Eth1/7 S202 vpc40 Use Platform Dependent commands to verify route for destination SWID is present in hardware On N7K, first attach to appropriate module via attach module x BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 FabricPath: what comes from where show fabricpath isis switch show fabricpath isis topology summary show fabricpath isis tree show fabricpath isis database mgroup detail show fabricpath mroute DRAP FabricPath IS-IS M2RIB MFDM IGMP L2FM Supervisor Engine show l2 multicast trees show ip igmp snooping groups show forwarding distribution l2 multicast [vlan <>] 7K M2FIB Hardware Drivers MTM Switch Table Other HW MAC Table I/O Module BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 FabricPath: Multidestination (Flood) S1 S2 FP Vlans S101# show fabricpath isis topology summary FabricPath IS-IS Topology Summary Fabricpath IS-IS domain: default MT-0 Configured interfaces: port-channel1 Ethernet6/27 Ethernet6/28 Max number of trees: 2 Number of trees supported: 2 Tree id: 1, ftag: 1, root system: 8478.ac0e.4742, 1 Tree id: 2, ftag: 2 [transit-traffic-only], root system: 8478.ac5b.2b42, 2 Ftag Proxy Root: 8478.ac0e.4742 S101 S102 S201 S202 vpc30 vpc40 A ES S100 B C ES S200 D S101# show fabricpath isis trees MT-0 Topology 0, Tree 1, Swid routing table 1, L1 via Ethernet6/27, metric 0 2, L1 via Ethernet6/27, metric , L1 via Ethernet6/27, metric , L1 via Ethernet6/27, metric , L1 via Ethernet6/27, metric , L1 via Ethernet6/27, metric 40 Repeat on each switch to map out complete forwarding tree (FTAG 1) S1# show fabricpath isis trees MT-0 Topology 0, Tree 1, Swid routing table 2, L1 via port-channel1, metric , L1 via Ethernet6/19, metric , L1 via Ethernet6/19, metric , L1 via Ethernet6/20, metric , L1 via Ethernet6/21, metric , L1 via Ethernet6/21, metric , L1 via Ethernet6/22, metric 40 Check the topology roots for each FTAG Map out the active links How to read: on which interface in given FTAG will this switch accept multidestination traffic from given switch Example: accept traffic from switch 100 on E6/19 in FTAG1 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 FabricPath: Multidestination (Flood) S1 S2 FP Vlans S101# show fabricpath mroute vlan 100 flood (vlan/100, *, *), Flood, uptime: 02:01:06, isis Outgoing interface list: (count: 5) Switch-id 1, uptime: 02:01:06, isis Switch-id 2, uptime: 02:01:06, isis Switch-id 102, uptime: 01:59:40, isis Switch-id 201, uptime: 02:01:06, isis Switch-id 202, uptime: 02:01:06, isis S101 S102 S201 S202 vpc30 vpc40 A ES S100 B C ES S200 D S101# show fabricpath mroute vlan 100 flood resolved (ftag/2, vlan/100, *, *), Flood, uptime: 02:01:32, isis Outgoing interface list: (count: 5) Interface Ethernet6/28, Switch-id 1, uptime: 02:01:31, isis Interface Ethernet6/28, Switch-id 2, uptime: 02:01:31, isis Interface Ethernet6/28, Switch-id 102, uptime: 02:00:07, isis Interface Ethernet6/28, Switch-id 201, uptime: 02:01:31, isis Interface Ethernet6/28, Switch-id 202, uptime: 02:01:31, isis Flood entry traffic that will be flooded to all active ports (minus receiving port) in a Vlan (remember about dynamic pruning) Ignore multiple appearances of the same interface (interface appears 1 per destination switch) (ftag/1, vlan/100, *, *), Flood, uptime: 02:01:32, isis Outgoing interface list: (count: 5) Interface Ethernet6/27, Switch-id 1, uptime: 02:01:31, isis Interface Ethernet6/27, Switch-id 2, uptime: 02:01:31, isis Interface Ethernet6/27, Switch-id 102, uptime: 02:00:07, isis Interface Ethernet6/27, Switch-id 201, uptime: 02:01:31, isis Interface Ethernet6/27, Switch-id 202, uptime: 02:01:31, isis BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 FabricPath: IP Multicast Remember RPF check S1 S2 FP Vlans S202# show ip igmp snooping groups vlan 100 Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port Vlan Group Address Ver Type Port list 100 */* - RF Eth1/7 RF Eth1/ v2 D Po40 S101 vpc30 S102 S201 S202 vpc40 S101# show fabricpath isis database mgroup detail egrep "LSPID Group 00-01" LSPID Seq Number Checksum Lifetime A/P/O/T S x xEA2C /0/0/1 Group-Address : IP Multicast : Vlan : 100 Groups : 1 Group : Sources : 0 S x xBD /0/0/1 Group-Address : IP Multicast : Vlan : 100 Groups : 1 Group : Sources : 0 S101# show fabricpath mroute vlan 100 (vlan/100, , ), uptime: 20:35:57, isis Outgoing interface list: (count: 2) Switch-id 201, uptime: 20:35:57, isis Switch-id 202, uptime: 20:35:57, isis A ES S100 B C ES S200 D Multicast Sender *,G from local IGMP snooping Local IGMP/snooping entries are redistributed into FP L2 multicast prune subtrees built on each FP switch S101 hashes multicast to FTAG 1 (remember vpc+ affinity) S101# show fabricpath mroute vlan 100 ftag 1 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 67 Multicast Receiver (ftag/1, vlan/100, , ), uptime: 20:47:34, isis Outgoing interface list: (count: 2) Interface Ethernet6/27, Switch-id 201, uptime: 22:26:18, isis Interface Ethernet6/27, Switch-id 202, uptime: 22:26:18, isis

68 FabricPath: IP Multicast S1 S2 FP Vlans QUIZ Both S201 and S202 receive multicast stream, who forwards out vpc 40? S101 vpc30 S102 S201 x S202 vpc40 S202# show vpc 40 vpc status id Port Status Consistency Reason Active vlans vpc+ Attrib Po40 up success success DF: Partial, FP MAC: S201# show fabricpath isis database detail S sec Affinity Affinity : Nickname: 200 Numgraphs: 1 Graph-id: 1 S201# show fabricpath isis database detail S sec Affinity Affinity : Nickname: 200 Numgraphs: 1 Graph-id: 2 A ES S100 B C ES S200 D Multicast Sender vpc+ in partial status which means multidestination traffic is load-balanced between vpc peers S201 has affinity for FTAG 1 S202 has affinity for FTAG 2 S201 will forward this frame Multicast Receiver BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 FabricPath: Hardware Multicast MAC Multicast MACs are stored differently from usual exx.xxxx F1 module-4# show hardware mac address-table vlan <vlan> vdc <vdc> fe <fe> FE Valid PI BD MAC Index... PV RD NN UC PI_E8 SWID SSWID LID ef x07ffb... 0x x000 0x000 0x07ffb ef x07ffb... 0x x000 0x000 0x07ffb ef x07ffb... 0x x000 0x000 0x07ffb ef x07ffb... 0x x000 0x000 0x07ffb Each mac appears twice: once per FTAG, use show hard internal forwarding table mac to find which is which F2 module-6# show hardware mac address-table vlan <vlan> vdc <vdc> fe <fe> FE Valid PI BD MAC Index Stat SW Modi Age... SWID SSWID LID ic fied Byte a 0x x x064 0x00b 0x d 0x x x0c8 0x000 0x f x07fd8 1 0x x000 0x000 0x07fd f x07fda 1 0x x000 0x000 0x07fda BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 Looking back in time show fabricpath isis internal event-history adjacency events related to adjacencies (up/down/etc) show fabricpath isis internal event-history urib FP events related to URIB updates (for example to see whole history for given switch ID) show fabricpath isis internal event-history events Overall FP event history: DRAP interactions, switch additions, removals, SPFrelated events show fabricpath isis internal event-history drap switch ID, FTAG related events BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 Tools

72 Troubleshooting Tools: Pong Pong can be equated to L2Ping + L2TraceRoute Depends on IEEE 1588v2 HW support F-series, N5500, and N6000 all support PTP, but N5K/N6K at present doesn t support pong Works by sending 2 types of packets: 1 packet to store timestamps at each hop and 2nd to collect stored timestamps S101# pong destination-swid 2 destination-mac 8478.ac5b.2b42 vlan 100 details Legend (*) - software delay(not hardware latency) (#) - reverse path (NA) - not available Hop System-mac (switch-id) Switching time (sec, nsec) ac-0e ( 101) ac-0e ( 1) ac-0e ( 1) ac-5b-2b-42 ( 2) ac-5b-2b-42 ( 2) ac-0e ( 1) ac-0e ( 1) ac-0e ( 101) Round trip time: 0sec nsec Egress from SWID 101 Ingress SWID 1 Egress SWID 1 Etc.. BRKDCT-3313 Send frame to SWID 2 (SysID of SWID 2 = 8478.ac5b.2b42) * By default, Frame sent on VLAN 1. Be sure to specify appropriate VLAN MACs that can be reached: - SysID or static Not supported over ECMP on F Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 Troubleshooting Tools: FPOAM FPOAM (Fabricpath Operations Administration and Management) is an effective tool set to monitor and diagnose data plane failures in FP networks. ping fabricpath traceroute fabricpath mtrace fabricpath 202# mtrace fabricpath ftag 2 repeat 1 Codes: '!' - success, 'Q' - request not sent, '.' - timeout, 'D' - Destination Unreachable, 'X' - unknown return code, 'V' - VLAN nonexistent, 'v' - VLAN in suspended state, 'm' - malformed request, 'C' - Cross Connect Error, 'U' - Unknown RBridge nickname, 'n' - Not AF, '*' - Success, Optional Tlv incomplete, 'I' - Interface not in forwarding state, 'S' - Service Tag nonexistent, 's' - Service Tag in suspended state, 'c' - Corrupted Data/Test S101 S102 S201 S202 Fabricpath mtrace for multicast ftag 2, vlan 1 Code SwitchId Interface State TotalTime ==================================================! 201 Rcvd on Eth1/2 fwd 3ms! 101 Rcvd on Eth1/2 fwd 4ms! 102 Rcvd on Eth1/2 fwd 4ms ES S100 ES S200 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 Troubleshooting Tools: FPOAM S1 S2 FP Vlans OAM Profiles can be used to replicate data plane packet and follow the forwarding path S101 S102 S201 S # traceroute fabricpath switch-id 100 profile 2 Codes: '!' - success, 'Q' - request not sent, '.' - timeout, 'D' - Destination Unreachable, 'X' - unknown return code, 'V' - VLAN nonexistent, 'v' - VLAN in suspended state, 'm' - malformed request, 'C' - Cross Connect Error, 'U' - Unknown RBridge nickname, 'n' - Not AF, '*' - Success, Optional Tlv incomplete, 'I' - Interface not in forwarding state, 'S' - Service Tag nonexistent, 's' - Service Tag in suspended state, 'c' - Corrupted Data/Test Sender handle: 14 Hop Code SwitchId Interface State TotalTime PathId ============================================================ 1! 2 Rcvd on Eth6/2 fwd 3ms 2! 100 Rcvd on Eth1/1 fwd 4ms A ES S100 B C ES S200 D 202# show run fabricpath section "oam profile 2" fabricpath oam profile 2 vlan 100 flow forward ether-type 0x800 ip source ip destination mac-address source mac-address destination protocol 1 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 Troubleshooting Tools: Counters S1 S2 FP Vlans S202(config)# ip access-list test-stats S202(config-acl)# statistics per-entry S202(config-acl)# permit ip host host S202(config-acl)# permit ip any any S202(config-acl)# interface ethernet 1/7 S202(config-if)# ip port access-group test-stats in S202(config-if)# end S202# show ip access-lists test-stats IPV4 ACL test-stats statistics per-entry 10 permit ip / /32 [match=0] 20 permit ip any any [match=0]! Sent 5000 frames S202# show ip access-lists test-stats IPV4 ACL test-stats statistics per-entry 10 permit ip / /32 [match=5000] 20 permit ip any any [match=0] S101 vpc30 S102 S201 A ES S100 B C ES S200 D Find the likely interface to receive packets (note multidestination traffic might follow different path sh fab isis trees) Configure ACL with statistics per-entry which explicitly matches traffic in question Attach ACL to ingress FP port as a PACL Check the counters Run test traffic Check the counters again Compare S202 vpc40 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 Troubleshooting Tools: Counters Find ingress interface & attach to respective linecard Find Ingress FE instance Configure statistics (use FE+1) Print statistics Run traffic Print statistics again note statistics are in HEX Compare S1# attach module 6 module-6# show hardware internal dev-port-map CARD_TYPE: 48 port 10G FP port PHYS MAC_0 L2LKP L3LKP QUEUE SWICHF module-6# test fabricpath unicast configure route-stats vdc 2 ftag 1 switchid 200 fe 5 table [mp sw] commit module-6# show fabricpath unicast route-stats vdc 2 ftag 1 switchid 200 fe VDC FTAG SwitchID SubSwitchID FE Adjacency Statistics 4 Eth6/ Eth6/ module-6# show fabricpath unicast route-stats vdc 2 ftag 1 switchid 200 fe VDC FTAG SwitchID SubSwitchID FE Adjacency Statistics 4 Eth6/ Eth6/ Use MP table to get per next-hop stat if there is >1 next-hop, else use SW table BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Troubleshooting Tools: Error/Drop Counters Usual datapath troubleshooting apply on N7K 7k# show hardware internal errors module 6 diff... send 2000 transit packets using ping with timeout k# show hardware internal errors module 6 diff < 1008 Self-forwarding check OSA drop > 1008 Self-forwarding check OSA drop show hardware internal errors often produces lengthy outputs, use diff to just see what has changed between 2 timed samples (with some test traffic in the middle) < 2514 Ingress packets marked with drop_oth sent to IB > 2514 Ingress packets marked with drop_oth sent to IB < 50 smallcnt DSWID/DSSWID miss and DCE frame, def-gw disabled > 50 smallcnt DSWID/DSSWID miss and DCE frame, def-gw disabled And on N5K/N6K N5K# sh platform fwm info pif e1/5 i stats cdce Eth1/5 pd: tx stats: bytes frames discard 0 drop 0 Eth1/5 pd: rx stats: bytes frames discard 0 drop 1650 Eth1/5 pd cdce_addr: switchid 30 sub-switchid 0, endnodeid 0 Eth1/5 pd cdce_addr: Mcast 0, locally-adm 1, OutOfOrder/don't learn 0 Eth1/5 pd cdce_addr: localid 5, pbp_idx 0 N5K# sh platform fwm info asic-errors 0 Printing non zero Carmel error registers: DROP_SRC_VLAN_MBR: res0 = res1 = 0 [12] DROP_CDCE_SW_TBL_RPF_MISS: res0 = 4 res1 = 0 [30] DROP_SRC_FTAG_BITMAP_MBR: res0 = 5 res1 = 0 [31] DROP_SRC_MASK_TO_NULL: res0 = res1 = 0 [44] PIF (physical interface) maintains RX/TX and drop counters Check if drops are non-zero & growing (also check the ASIC number) Use ASIC-errors command to get a breakdown of drop reasons (and see if any are growing in with test/ping traffic) BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 Troubleshooting Tools: ELAM When the going gets tough Embedded Logic Analyzer Module (ELAM) is an engineering tool that is used to look inside Cisco ASICs. ELAM is architecture specific and therefore will have different capabilities and different CLI syntax across different forwarding engines (FE). It is possible to use ELAM as a capturing tool to validate: 1. Was the packet received 2. On which interface/vlan did the packet arrive 3. What did the packet look like 4. How was the packet altered and where was it sent It is not intrusive ELAM is NOT a supported feature. It is a diagnostic tool designed for internal use. Anything and everything about it may change from version to version without any notice It can be used at a very granular level to troubleshoot a single traffic flow which can be an invaluable tool to network administrators. BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 Troubleshooting Tool: ELAM Workflow Identify the expected ingress Forwarding Engine (FE) Configure an ELAM trigger to capture specific frame Start the ELAM After ELAM triggers, display and analyze the data Once triggered data can be displayed and analyzed Typical ELAM challenges Identifying the correct capture point and trigger Understanding the captured data (for complex cases) BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 Troubleshooting Tools: ELAM Basics to know before performing an ELAM Data Bus (DBUS) and Result Bus (RBUS) The DBUS contains several platform specific internal fields along with the header information from a frame required to make the forwarding decision. We use the DBUS information to validate where the frame was received and basic data about the frame. The RBUS will contain information about the forwarding decision to help determine if the frame was altered and where it was sent. Local Target Logic (LTL) The LTL is an index used to represent a port or group of ports. The source LTL index and the destination LTL index tell us which port the frame was received and where it was sent. BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 Troubleshooting Tools: ELAM Example S1 S2 FP Vlans Packet from host <-> , expected ingress interface Eth6/19 on N7K-F2 linecard of S1 S1# attach module 6 Attaching to module 6... module-6# show hardware internal dev-port-map FRONT PANEL PORT TO ASIC INSTANCE MAP FP port PHYS MAC_0 L2LKP L3LKP QUEUE SWICHF module-6# elam asic clipper instance 4 module-6(clipper-elam)# layer2 module-6(clipper-l2-elam)# trigger dbus ipv4 ingress if source-ipv4- address destination-ipv4-address module-6(clipper-l2-elam)# trigger rbus ingress if trig module-6(clipper-l2-elam)# start module-6(clipper-l2-elam)# status L2 DBUS Triggered L2 RBUS Triggered Start the ELAM, send the traffic and wait for it to trigger S101 Eth6/19 is on FE instance 4 (code name clipper) vpc30 Configure a trigger specific to this source/destination IP S102 S201 A ES S100 B C ES S200 D Linecard M-series F1 F2 F3 S202 vpc40 L2/L3 ASIC name Eureka/Lamira Orion Clipper Flanker BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 Troubleshooting Tools: ELAM Example S1 S2 FP Vlans module-6(clipper-l2-elam)# show dbus <snip> port-id : 0x2 last-ethertype : 0x800 vlan : 0x64 destination-index : 0x0 source-index : 0x62 bundle-port : 0x0 status-is-1q : 0x1 trill-encap : 0x0 mac-in-mac-valid : 0x1 dtag-ttl : 0x20 recirc-acos : 0x0 dtag-ftag : 0x1 source-ipv4-address: destination-ipv4-address: mim-destination-mac-address: 0200.c mim-source-mac-address: b.ffff destination-mac-address d source-mac-address: a ODA (0c ) = OSA (064.0b.ffff) = module-6(clipper-l2-elam)# show rbus <snip> di-ltl-index : 0x65 l3-multicast-di : 0x0 source-index : 0x62 vlan-id : 0x64 dtag=ftag : 0x1 dtag-ttl : 0x1f mim-destination-mac-address: 0200.c mim-source-mac-address: b.ffff S101 vpc30 S102 S201 A ES S100 B C ES S200 D Frame received on VLAN 100 (0x64) from a source-index of 0x62 (next slide) mac-in-mac valid (this is a FP frame) dtag-ttl: fabricpath TTL of 32 (0x20) S202 vpc40 Frame transmitted on vlan 100 (0x64) to a destination index of 0x65 (next slide) dtag-ttl: fabricpath TTL decremented to 31 (0xf1) BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 Troubleshooting Tools: ELAM Example S1 S2 FP Vlans S1# show system internal pixm info ltl 0x62 Member info Type LTL PHY_PORT Eth6/19 S1# show system internal pixm info ltl 0x65 Member info Type LTL PHY_PORT Eth6/22 Get mapping of source index to physical port Get mapping of destination index to physical port S101 S102 S201 S202 vpc30 vpc40 A ES S100 B C ES S200 D ELAM confirms that frame was received on Eth6/19, VLAN 100 with an OSA of and ODA of ELAM also confirms that frame was forwarded out Eth6/22 on VLAN 100 with a decremented FP TTL BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 83

84 Troubleshooting Tools: show tech show tech fabricpath isis show tech fabricpath switch-id show tech fabricpath topology Neither of these include FP routes, macs or comprehensive forwarding related data. Collect these separately: show tech l2fm detail show tech l2fm l2dbg show tech forwarding l2 unicast show tech forwarding l2 multicast BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 Troubleshooting Example: Broken HSRP Problem statement: HSRP active & standby do not see each other in certain vlans. For example in vlan 1317 standby (S2) sees the active (S1), but on active standby is unknown. A number of vlans are affected. This is new deployment. Initial assessment: possible reason for HSRP router not seeing other router is HSRP hello packets not being received. In our case it is likely active router, not receiving hello packets from standby Quick debug on S1 confirms it only sends hellos in vlan 1317 S1# debug hsrp engine packet hello interface vlan :03:30 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip :03:31 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip :03:32 hsrp: Vlan1317[17/V4]: Hello out Active pri 100 ip and on S2 we see hellos being sent and received S2# debug hsrp engine packet hello interface vlan :03:30 hsrp: Vlan1317[17/V4]: Hello in from State Active pri 100 ip :03:30 hsrp: Vlan1317[17/V4]: Hello out Standby pri 50 ip :03:31 hsrp: Vlan1317[17/V4]: Hello in from State Active pri 100 ip :03:31 hsrp: Vlan1317[17/V4]: Hello out Standby pri 50 ip S1 S3 S2 S4 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 85

86 Troubleshooting Example: Broken HSRP Are the HSRP frames from S2 to S1 getting lost? S1# sh fabricpath load-balance multicast ftag-selected flow-type l2 dst-mac e src-mac c07.ac11 ether-type 800 vlan 1317 module 1... FTAG SELECTED IS : 1 S1 S3 E1/1 E1/1 S2 S4 S2# sh fabricpath load-balance multicast ftag-selected flow-type l2 dst-mac e src-mac c ether-type 800 vlan 1317 module 1... FTAG SELECTED IS : 2 Findings so far: Working and Non-working packets may follow different paths Time to look at the Trees S2# sh fabricpath isis topology summary MT-0 Configured interfaces: portchannel1 Ethernet1/1 Ethernet1/2 Number of trees: 2 Tree id: 1, ftag: 1, root system: , 2 Tree id: 2, ftag: 2, root system: , 4 S2# show fabricpath isis trees MT-0 Topology 0, Tree 1, Swid routing table 1, L1 via port-channel1, metric Topology 0, Tree 2, Swid routing table 1, L1 via Ethernet1/1, metric S1 S2 FTAG 1 traffic uses Po1 (peer-link) S1 S2 FTAG 2 traffic uses E1/1 (goes through S4) BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 86

87 Troubleshooting Example: Broken HSRP S4 is transit switch for HSRP S2 S1 traffic, hence we will not see packets in debug. We need to look at the data plane level if hello packet arrives/leaves. Options: SPAN, Counters, ELAM Let s try hardware counters S1 S3 E1/1 E1/1 S2 S4 CBL drops grow at about the rate of HSRP hellos. CBL stands for Color Blocking logic (or Vlan Blocking Logic). Essentially, hardware logic defining whether given port/vlan is blocking or forwarding packets. S4# show hardware internal errors module Device:Orion Fwding Driver Role:L2 Mod: 1 Last Thu Apr 11 11:11: Device Statistics Category :: ERROR Instance:0 ID Name Value Ports smallcnt Pkt dropped due to CBL Ingress packets marked with drop_oth sent to IB Root cause: Vlan missing from transit switch All FP vlans must be defined on all FP switches, otherwise there might be issues similar to this for flooded traffic. ISIS will prune off unnecessary flood traffic towards tree branches that do not have ports behind them. S4# show fabricpath mroute vlan 1317 ERROR: Vlan 1317 does not exist S4# show hardware internal errors module 1 diff...wait some seconds... S4# show hardware internal errors module 1 diff < 29 smallcnt Pkt dropped due to CBL > 29 smallcnt Pkt dropped due to CBL S4# show vlan id 1317 VLAN 1317 not found in current VLAN database BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 87

88 Troubleshooting: Common Pitfalls All FP Vlans must be present on all FP switches else multicast trees might not be correct TCNs not propagated to required FP or CE switches. Configure STP domain where TCNs need to be propagated. Else, connectivity might be broken after re-convergence until MACs age out or are relearned At power up or reload, CE-side comes up faster than FP-side L2GW Inconsistency, ensure that FP switches have been configured with superior priority before connecting to CE switches. BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 88

89 CLI cheatsheet Interfaces in FP mode show fabricpath isis interface [brief] ISIS adjacencies show fabricpath isis adjacency [detail] Root information for the trees show fabricpath isis topology summary RPF information for the trees show fabricpath isis trees OIFs for the trees show fabricpath mroute Affinity to Ftags show fabricpath isis database detail show system internal m2rib ftag Pong pong destination-swid <sw#> destination-mac <mac-address> vlan <vlan> count <#> [detail] BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 89

90 Summary Core Concepts Known Unicast Best path with ECMP, Rest Tree-balanced Control Plane ISIS in the core, STP / IGMP snooping at CE Data Plane MAC address table, SwitchID table, Tree table (RPF) Troubleshooting Understand what should be happening, verify what is happening, find a deviation, zoom in and repeat BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 90 90

91 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card. Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 91

92 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings Related sessions BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 92

93 Thank you

94