Layer 3 Forwarding and Troubleshooting Deep Dive on Nexus 9000/3000

Transcription

1

2 Layer 3 Forwarding and Troubleshooting Deep Dive on Nexus 9000/3000 Ambrish Mehta Principal Engineer (INSBU Engineering) Swami Narayanan Principal Engineer (INSBU Engineering)

3 Session Abstract Closer look at various Layer 3 forwarding options available on Nexus 95xx/93xx/31xx/32xx data center switches. Details on Layer 3 forwarding in multi chip and single chip hardware architecture. Flexibility offered by Forwarding Engine(ASIC) and how NX-OS is utilising it to meet different deployment needs. Empower Network Engineer/Architect to identify ideal routing mode for network deployment.

4 Related Sessions BRKDCT Nexus 9000 architecture Friday 2 to 4 PM Room # 211

5 Glossary of Terms Network Forwarding Engine (Trident2 ASIC) 2 DMOD DPORT UFT LC FM Network Forwarding Engine 2 (Tomahawk ASIC) Destination Module (ASIC Instance) Destination Port (Port within ASIC instance) Unified Forwarding Table Line card Module Fabric Module LPM Longest Prefix Match (e.g IPv4 : /31 to /0, IPv6 /127 to /0) ALPM ECMP Algorithmic Longest Prefix Match Equal Cost Multipath

6 Agenda Introduction to 95xx/93xx/31xx/32xx Layer 3 Forwarding Architecture /2 Forwarding Architecture NX-OS Routing Modes and Deployment Scenarios Walk Through of Commonly used Layer 3 Commands Q & A

7 Introduction to 95xx/93xx/31xx/32xx

8 Nexus 95xx/93xx Product Family Nexus9500 Series Nexus9300 Series Nexus 9500 Series Switches Nexus9504/Nexus9508/Nexus9516 Nexus 9300 Series Switches N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396PX N9K-C93128TX

9 Nexus 31xx/32xx Product Family Nexus 31xx Nexus 32xx Nexus 31xx Series Switches N3K-3132Q N3K-3164Q Nexus 32xx Series Switches N3K-3232-C N3K-3264-C

10 Layer 3 Forwarding Architecture

11 Layer 3 Forwarding Architecture Categories Modular Multi-Chip Modular switches (95xx) belongs to this category. LC and FM Chip operates independently. Fixed Multi-Chip 3164Q belongs to this category Virtual/Pseudo LC and FM operates independently. Fixed Single-Chip 93xx, 3132Q, 3232-C and 3264-C belongs to this category Single Chip is making all forwarding decisions.

12 Modular Multi-Chip: LC and FM Connectivity DMOD 100 DMOD 100 DMOD 100 DMOD 100 DMOD 100 DMOD 100 Fabric 1 Fabric 2 Fabric 3 Fabric 4 Fabric 5 Fabric 6 Fabric Module Interface DMOD DPORT Eth1/1 1 1 Eth1/ DMOD 1 DMOD 2 DMOD 3 36-port 40 Gbps QSFP+ Line Card/IO module HiGig Trunk (Port-Channel) : Network Forwarding Engine (Forwarding ASIC) Each of the 3 s on the line card has 12 x 40 Gbps links to fabric modules The above diagram shows the line card fabric connectivity in a Nexus 9508 chassis as an example LC and FM are connected via HiGig Trunk (Port-Channel) Each Front Panel port in the switch has unique identifier (DMOD/DPORT)

13 Modular Multi-Chip : Layer 3 Building Blocks Separate but identical is present on LC and FM We can make use of both to program routes and get higher scale Treat LC and FM s as independent forwarding entity Treat IPv4 Host, IPv4 LPM, IPv6 Host and IPv6 LPM as independent entity Identify programming location of prefixes blocks based on the Configured Routing Mode. Prefix Blocks IPv4 Host IPv4 LPM IPv6 Host IPv6 LPM

14 Modular Multi-Chip : Layer 3 Building Blocks IPv4 Host IPv6 Host IPv4 LPM IPv6 LPM Fabric Module IPv4 LPM Option 1 IPv6 LPM NX-OS Software Fabric Module IPv4 Host Option 2 IPv4 LPM Option 3 Fabric Module IPv4 Host Line Card IPv6 Host IPv6 Host Line Card IPv6 LPM IPv4 Host IPv6 Host IPv4 LPM Line Card IPv6 LPM

15 Modular Multi-Chip : Hierarchical Routing Fabric 1 Fabric 6 Fabric Modules Line Card/IO module 36-port 40 Gbps QSFP+ LC and FM both have same Use 2 Stage Forwarding Decision to efficiently utilise LC and FM tables (Higher Route Prefix Scale) LC handles 1 set of forwarding lookup, FM handles other set

16 Hierarchical Routing LC Lookup Hybrid Switching HiGig Header Fabric 1 Fabric 6 Fabric Modules 3 12 Data Packet LC HIT -> Egress DMOD, DPORT DMOD 1 DMOD 2 DMOD 3 Eth1/1 Eth1/36 Line Card/IO module LC forwarding decision HIT, Egress Point of Exit (DMOD, DPORT) derived Packet forwarded to Egress Interface directly. Incase Egress interface in different, then goes through FM (only Hybrid Switching and no forwarding lookup)

17 Hierarchical Routing FM Lookup FM Hit => Egress DMOD, DPORT HiGig Header Data Packet DMOD 100 DMOD 100 HiGig Header Fabric Modules 3 12 Data Packet LC Miss, Hit Catch All -> Punt to FM (DMOD 100) DMOD 1 DMOD 2 DMOD 3 Eth1/1 Eth1/36 Line Card/IO module LC Catch All Entry to punt packet to FM for lookup Special DMOD (DMOD 100) in HiGiG Header. FM does forwarding lookup if DMOD is 100 FM Lookup HIT Packet Forwarded to Egress Interface (DMOD, DPORT) FM Lookup NO HIT Packet punted from FM to SUP for ICMP Unreachable (Rate policed)

18 Non Hierarchical Routing Hybrid Switching Fabric 1 Fabric 6 Fabric Modules LC HIT -> Egress DMOD, DPORT Eth1/1 Eth1/36 Line Card/IO module Single Stage Forwarding Decision (only LC ) Fabric used for Hybrid Switching only Scale limited to Single Chip mode setting No HIT, packets punted to SUP for ICMP Unreachable (Rate policed)

19 Fixed Multi-Chip : Pseudo LC and FM Connectivity DMOD 100 Pseudo FM HiGig Trunk (Port-Channel) DMOD 1 DMOD 2 DMOD 3 DMOD 4 Pseudo LC 64-port 40 Gbps QSFP+ Same as 95XX Modular Chip Forwarding Architecture Use 2 Stage Forwarding Decision to efficiently utilise LC and FM tables (Higher Route Prefix Scale) LC handles 1 set of forwarding lookup, FM handles other set All modes supported in Fixed Multi-Chip operates same as Modular Multi Chip

20 Fixed Single-chip N3k-3232-C IPv4: Host and LPM Routes IPv6: Host and LPM Routes 2 32 x 100 Gbps All IPv4 and IPv6 host and LPM routes are programmed on this Single 2 Single ASIC needs to do all forwarding decision

21 Flexibility offered by /2

22 Unified Forwarding Table Bank Allocation Bank 0-1 Dedicated L2 Bank Banks 2-3 Configurable for MAC/Hosts/ LPM Banks 4-5 Configurable for MAC/Hosts/ LPM Bank 6-9 Dedicated L3 Hosts (V4 + V6 combined) 16K LPM Table for V4 and V6 LPM routes

23 UFT Mode 2 Bank 0-1 Dedicated L2 Bank Banks 2-3 Carved for Layer 2 MAC Banks 4-5 Carved for Layer 3 V4 & V6 Hosts Bank 6-9 Dedicated L3 Hosts (V4 + V6 combined) 16K LPM Table for V4 and V6 LPM routes

24 UFT Mode 3 Bank 0-1 Dedicated L2 Bank Bank 2 Carved for L2 Bank 3 Carved for L3 V4 + V6 Hosts Banks 4-5 Carved for Layer 3 V4 + V6 Hosts Bank 6-9 Dedicated L3 Hosts Hosts(V4 + V6 combined) 16K LPM Table for V4 and V6 LPM routes

25 TCAM LPM Table LPM Type # TCAM Entries LPM Scale V4 * 1 12K/4K V6 (<=/64) * 2 6K/2K IPv4/IPv6 (<=/64) 16K V6 (/65 to /127) 4 1K/3K** * Shared TCAM space between IPv4 LPM and IPv6 LPM <=/64 ** Default TCAM allocation allows1k prefixes IPv6 /65 to /127 IPv6 (/65 to /127) IPv6 /65 to /127 prefix length carving configuration N9k(config)# hardware profile ipv6 lpm-entries maximum < > Warning: The command will take effect after next reload. N9k(config)#

26 UFT Mode 4 Bank 0-1 Dedicated L2 Bank Banks 2-3 Carved for storing the V4/V6 LPM prefixes Banks 4-5 Carved for storing the V4/V6 LPM prefixes Bank 6-9 Dedicated L3 Hosts (V4 + V6 combined) 16K LPM Table (Indexing to Bank2-5)

27 ALPM and IPv6 Modes Algorithmic LPM - Repurposed UFT table to achieve higher LPM scale for IPv4 and IPv6 Supports two modes for IPv6 LPM (64B and 128B) IPv6 mode setting at level [not individual TCAM banks] 128B mode : All LPM prefix length uses128 bit width entries 64B mode : No /65 to /127 route can be programmed on 128B Mode : V6 LPM Scale Tunable with configuration

28 IPv6 ALPM Modes ALPM Mode 2 64B V4 LPM : 128K V6 LPM : 80K V4 LPM : 128K V6 LPM : 80K 128B V4 LPM : 128K/64K V6 LPM : 16K/20K * V4 LPM : 128K/64K V6 LPM : 10K/15K * * Achieved with carve-value configuration knob N9k(config)# hardware profile ipv6 alpm carve-value < > Warning: The command will take effect after next reload. N9k(config)#

29 Route Lookup and Rewrite Unicast Route Forwarding has 2 components Route Lookup Rewrite packet for next hop Rewrite information contains (Next Hop MAC, Egress Interface) ARP/Neighbor Discovery provides the information stores these details in Rewrite table (Egress Next Hop Table) Egress NH Table Table Size 48k/32k (/2)

30 UFT Scale Mode L2 L3 Host * LPM * 0 288K 16K 16k 1 224k 56k 16k 2 160k 144k 16k 3 96k 208k 16k 4 32k 16k 128k 2 Mode L2 L3 Host * LPM * 0 144K 8k 16k 1 104k 40k 16k 2 72k 72k 16k 3 40k 104k 16k 4 8k 8k 128k * Number of IPv4 Entries

31 NX-OS Routing Modes and Deployment Scenarios

32 Default Routing Mode: Multi-Chip Hierarchical Routing on 95xx/3164 switches. Suitable for most deployment scenarios. Leverages LC and FM for unicast route programming. Provides good balanced scale between Host and LPM prefixes for IPv4 and IPv6.

33 Nexus 950X/3164 Default Routing Mode: Hierarchical V4 & V6 Host Routes Forwarding Lookup performed in LC. Egress and Port derived Packets are sent directly to Egress (FM acts as bypass and it performs HiGig switching for this packet) V4 & V6 LPM Routes No host route match in LC LC LPM table has catch all default route to redirect the packets towards FM for LPM lookup FM performs the forwarding lookup to determine Egress and Port Data traffic forwarded to Egress IPv4 LPM IPv4 Host Fabric Module UFT Mode 4... Line Card UFT Mode 3 UFT Mode L3 Host LPM 3 208k 16k * 4 N/A 128k IPv6 LPM IPv6 Host * Catch All Default Route (0/0 and 0::/0) is programmed.

34 Nexus 950X/3164 : Default Mode - Host Lookup HiGiG Switching IPv4: /0 to /31 IPv6: /0 to /127 Fabric 1 Fabric 6 Fabric Modules IPv4: 0/0 IPv6: 0/0 Host Route Match -> Dst, Port IPv4: /32 IPv6: /128 Line Card/IO module N9K1#show ip route /32, ubest/mbest: 1/0, attached *via , [20/0], 01:44:03 N9K1# Packet In Dest : Packet Out

35 Nexus 950X/3164 : Default Mode LPM Lookup LPM Lookup => Dst, Port IPv4: /0 to /31 IPv6: /0 to /127 Fabric 1 Fabric 6 Fabric Modules Punt to FM for LPM lookup IPv4: 0/0 IPv6: 0/0 HOST Lookup: Fail LPM Lookup: System Default Route to FM IPv4: /32 IPv6: /128 Line Card/IO module N9K1#show ip route /24, ubest/mbest: 1/0, attached *via , [20/0], 01:44:03 N9K1# Packet In Dest: Packet Out

36 Scale Number : Hierarchical Routing Mode ASIC V4 Host V6 Host V4 LPM V6 LPM 208K (48k^) 104K 128K/64k 16k/20K K (32k^) 52K 128K/64k 10k/15K + N9k-1# show system routing mode Configured System Routing Mode: Default Applied System Routing Mode: Hierarchical (Default) Configured SVI post-routed unknown-unicast hardware flood mode: enabled N9k-1# + hardware profile ipv6 alpm carve-value <#> configuration knob ^ ARP/ND

37 Background and Requirements Default Hierarchical Mode is good for most deployments. Enhance NX-OS software to further utilize flexibility of Forwarding ASIC. Move around UFT Modes and IPv4/IPv6 prefix blocks to address different deployment needs. My Network has around IPv6 60k (<= /64 prefix length), Is there a way to get higher IPv6 LPM scale? My deployment is for L2/L3 boundary role and I have large number of hosts behind these Vlans. Is there a way to get higher ARP/ND scale? My Network s routing table size is very small, is there a way to avoid 2 stage look up?

38 Deployment Scenario 1 Large scale IPv6 LPM (/64) prefixes with some IPv4 LPM as well. Below is prefix scale with some growth room built in. IPv4: Host: <1k LPM: 2k IPv6: Host: <1k LPM: 60k (< /64) + 2k (/65 to /127)

39 Nexus 950X/3164 Hierarchical 64B ALPM V4 & V6 Host Routes Forwarding Lookup performed in LC, Egress and Port derived Packets are sent directly to Egress (FM acts as bypass and it performs HiGig switching for this packet) IPv4 Host Fabric Module UFT Mode 4 64Bit... Line Card UFT Mode 3 IPv6 Host UFT Mode L3 Host LPM 3 208k 16k 4 N/A 128k

40 Nexus 950X/3164 Hierarchical 64B ALPM V4 LPM & V6 <= /64 prefix length LPM Routes LC LPM table has default route to punt the packets towards FM for LPM lookup FM performs the forwarding lookup to determine Egress and Interface Data traffic forwarded to Egress V6 LPM prefix /65 to /127 Programmed in LC FM can take only V6 prefix length <= /64 IPv6 prefix length /65 to /127 programmed in LC in the LPM table. Hence scale limited to maximum of 3k IPv6 prefixes matching LPM prefixes send directly to Egress IPv4 LPM IPv4 Host Fabric Module UFT Mode 4 64Bit... IPv6 Line /65 Card to /127 UFT Mode LPM3 UFT Mode L3 Host LPM 3 208k 16k 4 N/A 128k IPv6 /64 LPM IPv6 Host

41 Nexus 950X/3164 : 64B ALPM - Host Lookup HiGiG Switching IPv4: /0 to /31 IPv6: /0 to /64 Fabric 1 Fabric 6 Fabric Modules IPv4: 0/0 IPv6: 0/0 Host Route Match -> Dst, Port N9K1#show ipv6 route 1:1:1.:1 1:1:1::1/128, ubest/mbest: 1/0, attached *via 4001::1, [20/0], 01:44:03 N9K1# IPv4: /32 IPv6: /128 IPv6: /65 to /127 Packet In Dest: 1:1:1::1 Packet Out Line Card/IO module

42 Nexus 950X/3164 : 64B ALPM LPM (V4 <=/31 and V6 <= /64) ALPM Lookup => Dst, Port IPv4: /0 to /31 IPv6: /0 to /64 Fabric 1 Fabric 6 Fabric Modules Punt to FM for LPM lookup IPv4: 0/0 IPv6: 0/0 HOST Lookup: Fail LPM Lookup: System Default Route to FM IPv4: /32 IPv6: /128, /65 to /127 Line Card/IO module N9K1#show ipv6 route 2:2:2::1 2:2:2::/48, ubest/mbest: 1/0, attached *via 2001::1, [20/0], 01:44:03 N9K1# Packet In Dest: 2:2:2::1 Packet Out

43 Nexus 950X/3164 : 64B ALPM V6 LPM (/65 to /127) HiGiG Switching IPv4: /0 to /31 IPv6: /0 to /64 Fabric 1 Fabric 6 Fabric Modules IPv4: 0/0 IPv6: 0/0 HOST Lookup: Fail LPM Lookup: Route Match -> Dst, Port IPv4: /32 IPv6: /128, /65 to /127 Line Card/IO module N9K1#show ipv6 route 3:3:3:3:3::3 3:3:3:3:3::/80, ubest/mbest: 1/0, attached *via 2002::1, [20/0], 01:44:03 N9K1# Packet In Dest: 3:3:3:3:3::3 Packet Out

44 Scale Number : 64B ALPM Mode ASIC V4 Host V6 Host V4 LPM V6 LPM (<=64) V6 LPM (65-127) 208K (48k^) 104K 128K 80K 1K-3K* 2 104K (32k^) 52K 128K 80K 1K-3K* N9k-1(config)# system routing hierarchical max-mode l3 64b-alpm Warning: The command will take effect after next reload. Note: This requires copy running-config to startup-config before switch reload. N9k-1# * hardware profile ipv6 lpm-entries maximum <#> configuration knob ^ ARP/ND

45 Deployment Scenario 2 L2/L3 aggregation role with large scale ARP and ND needs. Below is prefix scale with some growth room built in IPv4: Host(ARP): 60k LPM: 3k IPv6: Host(ND): 40k LPM: 1k

46 Nexus 950X Max Host Mode V4 Host & LPM Routes On LC only a V4 default route programmed. Redirects all V4 hosts & LPM traffic to FM FM does best prefix match for both host and LPM prefixes Packets are sent to Egress IPv4 Host Fabric Module UFT Mode 3... Line Card UFT Mode 2 IPv4 LPM Mode MAC L3 Host LPM 2 160k 144k 16k 3 N/A 208k 16k

47 Nexus 950X Max Host Mode V6 Host & LPM Routes V6 Hosts and LPM prefixes programmed in LC LC performs the forwarding lookup to determine Egress and Interface. Data traffic forwarded to Egress directly IPv4 Host IPv6 Host Fabric Module UFT Mode 3... Line Card UFT Mode 2 IPv4 LPM IPv6 LPM Mode MAC L3 Host LPM 2 160k 144k 16k 3 N/A 208k 16k

48 Modular Multi-Chip : Max Host Mode V6 Lookup HiGiG Switching IPv4: /0 to /32 Fabric 1 Fabric 6 Fabric Modules IPv4: 0/0 V6 Route Match on LC -> Dst and Port Line Card/IO module IPv6: /0 to /128 N9K1#show ipv6 route 1:1:1::1 1:1:1::/48, ubest/mbest: 1/0, attached *via 2001::1, [20/0], 01:44:03 N9K1# Packet In Dest: 1:1:1::1 Packet Out

49 Modular Multi-Chip : Max Host Mode V4 Lookup Host/LPM lookup for V4 -> Dst, Port IPv4: /0 to /32 Fabric 1 Fabric 6 Fabric Modules Punt to FM for LPM lookup V4 LPM Route -> System Default Route Match on LC N9K1#show ip route /32, ubest/mbest: 1/0, attached *via , [20/0], 01:44:03 N9K1# IPv6: /0 to /128 IPV4: 0/0 to FM Packet In Dest: Packet Out Line Card/IO module

50 Scale Number : Max Host Mode ASIC V4 Host V6 Host V4 LPM V6 LPM (<=64) V6 LPM (65-127) 208K (60K^) 104K 16K 6K 1K-3K* N9k-1(config)# system routing max-mode host Warning: The command will take effect after next reload. Note: This requires copy running-config to startup-config before switch reload. N9k-1# * hardware profile ipv6 lpm-entries maximum <#> configuration knob ^ ARP

51 Deployment Scenario 3 Network prefixes are hierarchical and aggregated. Below is prefix scale with some growth room built in. IPv4: Host: 16k LPM: 6k IPv6: Host: 8k LPM: 3k If possible, avoid two stage forwarding lookup.

52 Nexus 950X/3164 Non Hierarchical Routing Mode All V4/V6 Unicast forwarding decision done at LC V4/V6 Host & LPM Routes Forwarding Lookup performed in LC, Egress and Port derived for all L3 Unicast traffic (both Host and LPM routes) Packets are sent directly to Egress (FM acts as bypass and it performs HiGig switching for this packet) IPv4 Host Fabric Module UFT Mode X Line Card UFT Mode 3 IPv4 LPM... IPv6 LPM IPv6 Host UFT Mode L3 Host LPM 3 208k 16k

53 Nexus 950X/3164 :Non Hierarchical Any Lookup HiGiG Switching Fabric 1 Fabric 6 Fabric Modules Host/LPM Route Match -> Dst, Port IPv4: /0 to /32 IPv6: /0 to /128 Line Card/IO module Packet In Packet Out

54 Scale Number : Non Hierarchical Routing Mode ASIC V4 Host V6 Host V4 LPM V6 LPM (<=64) V6 LPM (65-127) 208K (48k^) 104K 12K/4K 6K/2K 1K-3K* 2 104K (32k^) 52K 12K/4K 6K/2K 1K-3K* N9k-1(config)# system routing non-hierarchical Warning: The command will take effect after next reload. Note: This requires copy running-config to startup-config before switch reload. N9k-1# * hardware profile ipv6 lpm-entries maximum <#> configuration knob ^ ARP/ND

55 Modular Multi-Chip Summary IPv4 Host IPv6 Host IPv4 LPM IPv6 LPM Fabric Module IPv4 LPM Default Mode IPv6 LPM NX-OS Software Fabric Module IPv4 Host Max-host Mode IPv4 LPM Non-Hierarchical Mode Fabric Module IPv4 Host IPv6 Host IPv6 Host IPv6 LPM IPv4 Host IPv6 Host IPv4 LPM IPv6 LPM Line Card Line Card Line Card

56 Single Chip Routing Modes

57 Nexus 93xx/31xx/32xx Default Routing IPv6 LPM IPv6 Host UFT Mode 3 IPv4 LPM IPv4 Host Mode MAC L3 Host LPM 3 96k 208k 16k

58 Scale Number : 93xx/31xx/32xx Default Mode ASIC V4 Host V6 Host V4 LPM V6 LPM (<=64) N9k-1# show system routing mode Configured System Routing Mode: Default Applied System Routing Mode: Max Host (Default) N9k-1# V6 LPM (65-127) 208K (48k^) 104K 12K/4K 6K/2K 1K-3K* 2 104K (32k^) 52K 12K/4K 6K/2K 1K-3K* * hardware profile ipv6 lpm-entries maximum <#> configuration knob ^ ARP/ND

59 Nexus 93xx/31xx/32xx Max L3 Mode IPv6 LPM IPv6 Host UFT Mode 4 IPv4 LPM IPv4 Host Mode MAC L3 Host LPM 4 32k 16k 128k

60 Scale Number: 93xx/31xx/32xx Max L3 ASIC V4 Host V6 Host V4 LPM V6 LPM (0-127) 16K 8K 128K 16K-20K + 2 8K 4K 128K 10K-15K + N9k-1(config)# system routing max-mode l3 Warning: The command will take effect after next reload. Note: This requires copy running-config to startup-config before switch reload. N9k-1# + hardware profile ipv6 alpm carve-value <#> configuration knob

61 Walk Through of Commonly used CLIs

62 System Routing Mode N9K1# show system routing mode Configured System Routing Mode: Default Applied System Routing Mode: Hierarchical (Default) Configured SVI post-routed unknown-unicast hardware flood mode: enabled N9K1# conf terminal Enter configuration commands, one per line. End with CNTL/Z. N9K1#(config)# system routing non-hierarchical Warning: The command will take effect after next reload. Note: This requires copy running-config to startup-config before switch reload. N9K1#(config)# end N9K1# show system routing mode Switch is not Configured System Routing Mode: Non-Hierarchical reloaded after Applied System Routing Mode: Hierarchical (Default) mode change Configured SVI post-routed unknown-unicast hardware flood mode: enabled N9K1#

63 IPv4 Host Route With Single Nexthop Example N9K1#show ip route /32, ubest/mbest: 1/0, attached /32 Host Route *via , [20/0], 01:44:03, bgp-300, external, tag N9K1#show forwarding ip route /32 module 1 IPv4 routes for table default/base Prefix Next-hop Interface Labels Partial Install * / Ethernet1/1 NextHop Physical Interface

64 Forwarding Routing Modes Verification N9K1#show forwarding ip route summary module 1 Module Type : Line-Card Module Mode : Mode-3 Module Route Download-type : Host only (IPv4+IPv6) (1) N9K1#show forwarding ip route summary module 22 Module Type : Fabric-Module Module Mode : ALPM (Mode-4) Module Route Download-type : LPM only (IPv4+IPv6) (2) N9K1# Line Card is running in Mode 3 Fabric Module is running in Mode 4

65 bcm-shell command reference Command Usage l3 l3table show Display IPv4 Host Entries l3 defip show l3 ip6host show l3 ip6route show l3 multipath show l3 egress show show c rpkt show c tpkt Display IPv4 LPM Entries Display IPv6 Host Entries Display IPv6 LPM Entries Display ECMP object and Nexthop List Display Egress object with rewrite information Display Received packets in Hardware Display Transmitted packets in Hardware

66 IPv4 Host Route With Single Nexthop Example N9K1#bcm-shell module 1 "0:l3 l3table show" inc :00:00:00:00: n Egress Object N9K1#bcm-shell module 1 "0:l3 egress show " Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC :73:5c:fe:e4:6e no no no Nexthop MAC Unique hardware port identifier N9K1#show system internal ethpm info interface ethernet 1/1 grep -i STATIC IF_STATIC_INFO: port_name=ethernet1/1,if_index:0x ,ltl=5952,slot=0, nxos_port=48,dmod=2,dpid=13,unit=1,queue=2160,xbar_unitbmp=0x2,ns_pid=255,slice_num =0,port_on_slice=0,src_id=0

67 IPv4 LPM Route With ECMP Example N9K1#show ip route /24, ubest/mbest: 2/0, attached /24 LPM Route *via , [20/0], 01:44:03, bgp-300, external, tag *via , [20/0], 01:44:04, bgp-300, external, tag N9K1#show forwarding ip route /24 module 21 IPv4 routes for table default/base NextHop Physical Interfaces Prefix Next-hop Interface Labels Partial Install * / Ethernet1/ Ethernet1/2

68 IPv4 LPM Route With ECMP Example N9K1#bcm-shell module 21 "0:l3 defip show" inc / /24 00:00:00:00:00: n ECMP Object N9K1#bcm-shell module 21 "0:l3 multipath show " Multipath Egress Object Interfaces: Egress Objects N9K1#bcm-shell module 21 "0:l3 egress show " Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC :73:5c:fe:e4:6e no no no N9K1#bcm-shell module 21 "0:l3 egress show " Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC :73:5c:fe:e4:6e no no no

69 Layer 3 Consistency Checker N9K1# test forwarding inconsistency module 1 Consistency check started. N9k-1# show forwarding inconsistency module 1 IPV4 Consistency check : table_id(0x1) slot(1) Execution time : ms () No inconsistent adjacencies. No inconsistent routes. Consistency-Checker: PASS for 1 N9k-1# test forwarding ipv6 inconsistency module 1 V6 Consistency check started. N9k-1# show forwarding ipv6 inconsistency module 1 IPV6 Consistency check : table_id(0x ) slot(1) Execution time : 3130 ms () No inconsistent adjacencies. No inconsistent routes. Consistency-Checker: PASS for 1 N9k-1#

70 Q & A

71 Complete Your Online Session Evaluation Give us your feedback and receive a Cisco 2016 T-Shirt by completing the Overall Event Survey and 5 Session Evaluations. Directly from your mobile device on the Cisco Live Mobile App By visiting the Cisco Live Mobile Site Visit any Cisco Live Internet Station located throughout the venue T-Shirts can be collected Friday 11 March at Registration Learn online with Cisco Live! Visit us online after the conference for full access to session videos and presentations.

72 Thank you

73