Cisco Nexus 7000 Hardware Architecture BRKARC-3470

Transcription

1 Cisco Nexus 7000 Hardware Architecture BRKARC-3470

2 Session Goal To provide you with a thorough understanding of the Cisco Nexus 7000 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions This session will not examine Unified I/O, DCB, FCoE, NX- OS software architecture, or other Nexus platforms Related sessions: BRKARC-3471: Cisco NXOS Software Architecture 2 2

3 What Is Nexus 7000? Data-center class Ethernet switch designed to deliver high-availability, system scale, usability, investment protection Supervisor s I/O Modules Chassis Fabrics Forwarding s 3

4 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 4

5 Nexus 7010 Chassis System status LEDs ID LEDs on all FRUs Front-toback airflow Integrated cable management with cover Air exhaust Optional locking front doors System fan trays Locking ejector levers Supervisor slots (5-6) I/O module slots (1-4, 7-10) 21RU Fabric fan trays Two chassis per 7 rack Crossbar fabric modules Power supplies Air intake with optional filter Front N7K-C7010 Rear Common equipment removes from rear 5

6 Integrated cable management Nexus 7018 Chassis System status LEDs Optional front door Supported in NX-OS release 4.1(2) and later System fan trays Side-to-side airflow Supervisor slots (9-10) 25RU Crossbar fabric modules I/O module slots (1-8, 11-18) Common equipment removes from rear Power supply air intake Power supplies Front N7K-C7018 Rear 6

7 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 7

8 Supervisor Performs control plane and management functions Dual-core 1.66GHz x86 processor with 8GB DRAM 2MB NVRAM, 2GB internal bootdisk, compact flash slots, USB Console, aux, and out-of-band management interfaces Interfaces with I/O modules via 1G switched EOBC Houses dedicated central arbiter ASIC that controls VOQ admission/fabric access via dedicated arbitration path to I/O modules N7K-SUP1 ID LED Status LEDs AUX Port Console Port Management Ethernet Compact Flash Slots Reset Button USB Ports CMP Ethernet 8

9 Nexus 7000 I/O Module Families M1 and F1 M family L2/L3/L4 with large forwarding tables and rich feature set N7K-M148GT-11/N7K-M148GT-11L N7K-M132XP-12/ N7K-M132XP-12L N7K-M108X2-12L N7K-M148GS-11/N7K-M148GS-11L F family Low-cost, high performance, low latency, low power and streamlined feature set N7K-F132XP-15 9

10 8-Port 10GE M1 I/O Module N7K-M108X2-12L Supported in NX-OS release 5.0(2a) and later 8-port 10G with X2 transceivers 80G full-duplex fabric connectivity Two integrated forwarding engines (120Mpps) Support for XL forwarding tables (licensed feature) 8 ports wire-rate L3 multicast replication 802.1AE LinkSec N7K-M108X2-12L 11

11 EOBC 8-Port 10G XL M1 I/O Module Architecture N7K-M108X2-12L To Central Arbiter To Fabric Modules Fabric ASIC LC CPU VOQs Forwarding Forwarding VOQs Replication Replication Replication Replication 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC Linksec Linksec Linksec Linksec Linksec Linksec Linksec Linksec Front Panel Ports 12

12 N7K-M132XP-12L Supported in NX-OS release 5.1(1) and later 32-Port 10GE M1 I/O Modules N7K-M132XP-12, N7K-M132XP-12L 32-port 10G with SFP+ transceivers 80G full-duplex fabric connectivity Integrated 60Mpps forwarding engine XL forwarding engine on L version Oversubscription option for higher density (up to 4:1) Supports Nexus 2000 (FEX) connections 8 ports wire-rate L3 multicast replication 802.1AE LinkSec N7K-M132XP-12/ N7K-M132XP-12L BRKARC Cisco and/or its affiliates. Cisco All Public rights reserved. Cisco Public 13

13 Shared vs. Dedicated Mode To Fabric rate-mode shared (default) 10G Shared mode Four interfaces in port group share 10G bandwidth To Fabric rate-mode dedicated 10G Port group group of contiguous even or odd ports that share 10G of bandwidth (e.g., ports 1,3,5,7) Dedicated mode First interface in port group gets 10G bandwidth Other three interfaces in port group disabled 14

14 EOBC 32-Port 10G M1 I/O Module Architecture N7K-M132XP-12, N7K-M132XP-12L To Central Arbiter To Fabric Modules Fabric ASIC LC CPU VOQs Forwarding VOQs Replication Replication Replication Replication 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 4:1 Mux + Linksec 4:1 Mux + Linksec 4:1 Mux + Linksec 4:1 Mux + Linksec :1 Mux + Linksec 4:1 Mux + Linksec 4:1 Mux + Linksec 4:1 Mux + Linksec Front Panel Ports 15

15 48-Port 1G M1 I/O Modules N7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L Four 1G I/O module options: 48 10/100/1000 RJ-45 ports (N7K-M148GT-11) 48 1G SFP ports (N7K-M148GS-11) 48 1G SFP ports with XL forwarding engine (N7K-M148GS-11L) 48 10/100/1000 RJ-45 ports with XL forwarding engine (N7K-M148GT-11L) Integrated 60Mpps forwarding engine 46G full duplex fabric connectivity Line rate on 48-ports with some local switching 48 ports wire-rate L3 multicast replication 802.1AE LinkSec N7K-M148GT-11 Release 4.0(1) and later N7K-M148GS-11 Release 4.1(2) and later N7K-M148GS-11L Release 5.0(2a) and later N7K-M148GT-11L Release 5.1(1) and later 16

16 48-Port 1G M1 I/O Modules Architecture N7K-M148GT-11, N7K-M148GS-11, N7K-M148GS-11L, N7K-M148GT-11L EOBC To Fabric Modules To Central Arbiter Fabric ASIC LC CPU VOQs Replication Forwarding Replication 12 x 1G MAC 12 x 1G MAC 12 x 1G MAC 12 x 1G MAC Linksec Linksec Linksec Linksec Linksec Linksec Front Panel Ports 17

17 32-Port 1G/10GE F1 I/O Module N7K-F132XP-15 Supported in NX-OS release 5.1(1) and later 32-port 1G/10G with SFP/SFP+ transceivers 230G full-duplex fabric connectivity (320G local switching) System-on-chip (SoC) forwarding engine design 16 independent SoC ASICs Layer 2 forwarding with L3/L4 services (ACL/QoS) Multi-protocol Classic Ethernet, FabricPath, DCB, FCoE (future) N7K-F132XP-15 sometimes called switch-on-chip 18

18 32-Port 1G/10G F1 I/O Module Architecture N7K-F132XP-15 EOBC To Fabric Modules To Central Arbiter LC CPU Arbitration Aggregator Fabric ASIC Fabric ASIC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC Front Panel Ports 19

19 I/O Module Design Principles M1 I/O modules: Multi-chipset design Feature-rich Large forwarding tables and deep buffers in external memory F1 I/O modules: Integrated SoC design Streamlined feature set On-chip forwarding tables and buffering (no external memory interfaces) 20

20 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 21

21 M1 Forwarding Hardware Hardware forwarding engine(s) integrated on every I/O module 60Mpps per forwarding engine Layer 2 bridging with hardware MAC learning 60Mpps per forwarding engine IPv4 and 30Mpps IPv6 unicast IPv4 and IPv6 multicast support (SM, SSM, bidir) RACL/VACL/PACLs QoS remarking and policing policies Policy-based routing (PBR) Unicast RPF check and IP source guard Ingress and egress NetFlow (full and sampled) Hardware Table M1 Modules M1-XL Modules without License M1-XL Modules with License FIB TCAM 128K 128K 900K Classification TCAM (ACL/QoS) 64K 64K 128K MAC Address Table 128K 128K 128K NetFlow Table 512K 512K 512K 22

22 M1 Forwarding Architecture FE Daughter Card Ingress NetFlow collection Ingress ACL and QoS classification lookups Layer 3 Ingress Pipeline Egress Pipeline Egress NetFlow collection Egress policing FIB TCAM and adjacency table lookups for Layer 3 forwarding ECMP hashing Multicast RPF check Ingress policing Unicast RPF check Ingress MAC table lookups IGMP snooping lookups IGMP snooping redirection Egress ACL and QoS classification lookups Layer 2 Egress MAC lookups IGMP snooping lookups Packet Headers from Final lookup result to I/O Module Replication I/O Module Replication 23

23 F1 Forwarding Hardware SoC forwarding engine services two front-panel 10G ports 16 SoCs per I/O module 480Mpps Layer 2 bridging with hardware MAC learning per I/O module 30Mpps per forwarding engine VACL/PACLs QoS remarking policies FabricPath forwarding Priority Flow-Control (PFC) and Enhanced Transmission Selection (ETS) Hardware Table Per SoC Per F1 Module MAC Address Table 16K 256K Classification TCAM (ACL/QoS) 1K in/1k out 16K in/16k out 24

24 F1 Forwarding Architecture To/From Central Arbiter To Fabric Virtual output queues From Fabric Egress forwarding decision (egress ACL FCoE only) Ingress forwarding decision (MAC lookup, ingress ACL/QoS Ingress Buffer (VOQ) Forwarding (Ingress) MAC Table, ACL, QoS Forwarding (Egress) Egress Buffer 2 X 10G SoC Egress fabric receive buffer Skid buffer Accommodates pause reaction time Pre-Forwarding Ingress Buffer Forwarding tables 1G and 10G capable interface MAC 1G/10G MAC 1G/10G MAC Port A 1G/10G Two front-panel interfaces per ASIC Port B 1G/10G 25

25 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 26

26 Crossbar Switch Fabric Module Each fabric module provides 46Gbps per I/O module slot Up to 230Gbps per slot with 5 fabric modules Different I/O modules leverage different amount of fabric bandwidth 80G per slot with 10G M1 modules 230G per slot with 10G F1 modules Access to fabric controlled using QoS-aware central arbitration with VOQ N7K-C7010-FAB-1 N7K-C7018-FAB-1 27

27 Multistage Crossbar Nexus 7000 implements 3-stage crossbar switch fabric Stages 1 and 3 on I/O modules Stage 2 on fabric modules Fabric Modules 2nd stage 1 Crossbar Fabric ASIC 2 Crossbar 3 Crossbar 4 Crossbar 5 Fabric ASIC Fabric ASIC Fabric ASIC Crossbar Fabric ASIC 2 x 23Gbps per I/O slot per fabric module Up to 230Gbps per I/O module with 5 fabric modules installed 1st stage Crossbar Fabric ASIC Ingress Module Crossbar Fabric ASIC Egress Module 3rd stage 20 x 23Gbps channels per fabric module 28

28 Fabric Module Capacity 2 x 23G channels per I/O module slot 46Gbps/slot Fabric Modules Crossbar Fabric ASICs 1 1 x 23G channel per supervisor slot 46Gbps/slot Crossbar Fabric ASICs 2 184Gbps 138Gbps 230Gbps 46Gbps 92Gbps per slot bandwidth 46Gbps/slot 46Gbps/slot Crossbar Fabric ASICs Crossbar Fabric ASICs Gbps/slot Crossbar Fabric ASICs 5 Nexus

29 M1 I/O Module Capacity 1G modules Require 1 fabric for full bandwidth Require 2 fabrics for N+1 redundancy 46Gbps/slot 46Gbps/slot Fabric Modules Crossbar Fabric ASICs Crossbar Fabric ASICs Gbps 138Gbps 230Gbps 46Gbps 92Gbps per slot bandwidth 46Gbps/slot Crossbar Fabric ASICs 3 4th and 5th fabric modules provide additional redundancy and future-proofing 46Gbps/slot Crossbar Fabric ASICs 4 10G modules Require 2 fabrics for full bandwidth Require 3 fabrics for N+1 redundancy 46Gbps/slot Crossbar Fabric ASICs 30 5

30 F1 I/O Module Capacity F1 SFP+ module Operates with any number of fabrics Requires 5 fabrics for maximum bandwidth Redundancy model is graceful bandwidth derating 230G 46Gbps/slot 46Gbps/slot Fabric Modules Crossbar Fabric ASICs Crossbar Fabric ASICs Gbps 138Gbps 230Gbps 46Gbps 92Gbps per slot bandwidth 46Gbps/slot Crossbar Fabric ASICs 3 230G 46Gbps/slot Crossbar Fabric ASICs 4 46Gbps/slot Crossbar Fabric ASICs 5 31

31 Access to Fabric Bandwidth Access to fabric controlled using central arbitration Arbiter ASIC on Supervisor provides fabric arbitration via dedicated arbitration path Virtual Output Queues (VOQs) at ingress to fabric represent bandwidth availability on egress modules Four levels of priority per VOQ destination Central arbiter controls admission to VOQ based on bandwidth availability and priority Buffer credits represent bandwidth availability for each VOQ destination at each priority level Credits requested by ingress I/O modules with traffic to send into fabric Credits granted by central arbiter based on bandwidth availability Credits returned to the pool by egress I/O modules after receiving traffic from fabric 32

32 Benefits of Central Arbitration with VOQ Ensures priority traffic takes precedence over besteffort traffic across fabric Four levels of priority for each VOQ destination Ensures fair access to bandwidth for multiple ingress ports transmitting to one egress port Central arbiter ensures all traffic sources get appropriate access to fabric bandwidth, even with traffic sources on different modules Prevents congested egress ports from blocking ingress traffic destined to other ports Mitigates head-of-line blocking by providing independent queues for individual destinations across the fabric Enables lossless service for some traffic classes across the fabric Can provide strict priority and backpressure (blocking instead of dropping) for certain traffic classes, such as FCoE traffic 34 34

33 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 35

34 I/O Module Queuing All configuration through Modular QoS CLI (MQC) Queuing parameters applied using class-maps/policymaps/service-policies Queuing and scheduling behavior based on I/O module capabilities Default queuing on all modules enables two classes including egress priority queue Buffering model varies by I/O module family M1 modules: hybrid model combining egress port buffered with ingress VOQ buffered architecture F1 modules: pure ingress VOQ buffered architecture 36

35 Hybrid Ingress/Egress Buffered Model (M1 I/O Modules) Ingress port buffer manages congestion in ingress forwarding/replication engines only Ingress VOQ buffer manages congestion toward egress interface over fabric Egress FIFO buffer just enough to catch frames in flight and keep pipe full Egress port buffer manages congestion at egress interface Ingress port buffer Ingress VOQ buffer Ingress Module Egress FIFO buffer Egress port buffer Ingress Module Crossbar Fabric Egress Module Ingress Module 37

36 Ingress Buffered Model (F1 I/O Modules) Ingress skid buffer just enough to absorb packets received after external flow control asserted Ingress VOQ buffer manages congestion toward egress interfaces Egress FIFO buffer just enough to catch frames in flight and keep pipe full Ingress skid buffer Ingress VOQ buffer Ingress Module Egress FIFO buffer Ingress Module Crossbar Fabric Egress Module Ingress Module 38

37 M1 I/O Module Buffering Hybrid ingress/egress buffered architecture on 8-port 10G M1 I/O module Fabric ASIC p3q1t Egress 80MB VOQ 16MB Shared VOQs Replication LC CPU Replication VOQs Replication Forwarding Fabric ASIC Forwarding Replication 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC 10G MAC VOQs Replication Linksec Linksec Linksec Linksec Linksec Linksec Linksec Linksec 1p7q4t Front Panel Ports Per-Port 10G MAC 10G MAC Ingress 96MB Port 1 Port 2 8q2t 40

38 F1 I/O Module Buffering Pure ingress buffered architecture on 32-port 10G F1 I/O module 1 FIFO Egress 750KB Fabric ASIC LC CPU Fabric ASIC Fabric ASIC Arbitration Aggregator p3q1t VOQ 1.25MB Per-Port 2 X 10G SoC 2 X 10G SoC 1 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC Front Panel Ports 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC 2 X 10G SoC Skid 250KB FIFO Port 1 Port 2 41

39 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 43

40 Layer 2 Forwarding MAC table size depends on module type: M1 MAC table is 128K entries F1 MAC table is 256K entries (16K entries per SoC) Hardware MAC learning CPU not directly involved in learning Forwarding engine(s) on each module have copy of MAC table New learns communicated to other forwarding engines via hardware flood to fabric mechanism Software process ensures continuous MAC table sync Spanning tree (PVRST or MST), Virtual Port Channel (VPC), or FabricPath ensures loop-free Layer 2 topology 44

41 Layer 2 Forwarding Architecture Layer 2 Forwarding Manager (L2FM) maintains central database of MAC tables L2FM Supervisor L2FM keeps MAC table on all forwarding engines in sync L2FM-Client process on I/O modules interfaces between L2FM and hardware MAC table n7010# sh processes cpu egrep PID l2fm PID Runtime(ms) Invoked usecs 1Sec Process l2fm n7010# attach mod 9 Attaching to module 9... To exit type 'exit', to abort type '$.' Last login: Mon Apr 21 15:58: from sup02 on pts/0 Linux lc _mvl401-pc_target #1 Fri Mar 21 23:26:28 PDT 2009 ppc GNU/Linux module-9# sh processes cpu egrep l2fm l2fmc module-9# L2FM-C L2FM-C Hardware Hardware I/O Module I/O Module Hardware MAC Table Hardware MAC Learning L2FM-C Hardware I/O Module 45

42 Hardware Layer 2 Forwarding Process In Classic Ethernet and FabricPath edge switches, MAC table lookup drives Layer 2 forwarding Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs Source MAC lookup drives new learns and refreshes aging timers Destination MAC lookup dictates outgoing switchport (CE/FabricPath local) or destination Switch ID (FabricPath remote) In FabricPath core switches, Switch ID (routing) table lookup drives Layer 2 forwarding Destination SID lookup dictates outgoing FabricPath interface and next hop 46

43 HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling M1 L2 Packet Flow Credit grant for fabric access 9 Supervisor Central Arbiter Return credit to pool 12 Fabric Module 1 Fabric Module 2 Fabric Module 3 VOQ arbitration and queuing 8 Transmit to fabric 10 Fabric ASIC Fabric ASIC Layer 3 Fabric ASIC ACL/QoS/ NetFlow lookups 6 Layer 3 Fabric ASIC Fabric ASIC Receive from fabric Return buffer credit 11 Submit packet headers for lookup 2nd stage ingress port QoS Receive packet from wire e1/1 VOQs Replication 10G MAC 4:1 Mux + Linksec 2 Layer 2 Forwarding LinkSec decryption 1 st stage ingress port QoS Module 1 L2 SMAC/ DMAC lookups Return result 5 7 Layer 2 Forwarding Module 2 Transmit packet on wire Egress port QoS VOQs Replication 10G MAC Linksec e2/1 LinkSec encryption 14

44 HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling F1 L2 Packet Flow Credit grant for fabric access 4 Supervisor Central Arbiter Return credit to pool 7 Fabric Module 1 Fabric ASIC Fabric Module 2 Fabric ASIC Fabric Module 3 Fabric ASIC VOQ arbitration and queuing 3 Receive packet from wire 1 5 Transmit to fabric e1/1 Fabric ASIC SoC e1/2 2 Module 1 Ingress L2 & ACL/QoS lookups 8 Transmit packet on wire e2/1 Fabric ASIC SoC e2/2 Receive from fabric Return buffer credit Module

45 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 49

46 IP Forwarding Nexus 7000 decouples control plane and data plane Forwarding tables built on control plane using routing protocols or static configuration OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing Tables downloaded to forwarding engine hardware for data plane forwarding 50

47 IP Forwarding Architecture Routing protocol processes learn routing information from neighbors IPv4 and IPv6 unicast RIBs calculate routing/next-hop information Unicast Forwarding Distribution Manager (UFDM) interfaces between URIBs on supervisor and IP FIB on I/O modules IP FIB process programs forwarding engine hardware on I/O modules FIB TCAM contains IP prefixes Adjacency table contains next-hop information n7010# sh processes cpu egrep ospf PID PID Runtime(ms) Invoked usecs 1Sec Process ospf n7010# sh processes cpu egrep u.?rib u6rib urib n7010# sh processes cpu egrep ufdm ufdm module-9# sh processes cpu egrep fib ipfib module-9# Supervisor BGP OSPF ISIS RIP EIGRP URIB/U6RIB UFDM IP FIB IP FIB IP FIB Hardware Hardware Hardware I/O Module I/O Module I/O Module Hardware FIB TCAM ADJ Table 51

48 Hardware IP Forwarding Process FIB TCAM lookup based on destination prefix (longest-match) FIB hit returns adjacency, adjacency contains rewrite information (next-hop) Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result 52

49 IPv4 FIB TCAM Lookup HIT! Generate Lookup Key xx xx xx xx xx xx xx xx FIB TCAM Generate TCAM lookup key (destination IP address) Compare lookup key Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Hit in FIB returns result in FIB DRAM Index, # next-hops FIB DRAM Ingress unicast IPv4 packet header Load-Sharing Hash # nexthops Flow Data Adj Index Adjacency index identifies ADJ block Offset Hash selects exact next hop entry Forwarding Next-hop 1 (IF, MAC) Next-hop 2 (IF, MAC) Next-hop 3 (IF, MAC) Next-hop 4 (IF, MAC) Next-hop 5 (IF, MAC) Next-hop 6 (IF, MAC) Next-hop 7 (IF, MAC) Adjacency Table Return lookup result Result 53

50 Routing vs. Forwarding Routing information refers to unicast RIB contents in supervisor control plane Forwarding information refers to FIB contents at I/O module 54

51 Displaying Routing and Forwarding Information show routing [ipv4 ipv6] [<prefix>] [vrf <vrf>] Displays software routing (URIB) information Can also use traditional show ip route command show forwarding [ipv4 ipv6] route module <mod> [vrf <vrf>] Displays hardware forwarding (FIB) information on permodule basis show forwarding adjacency module <mod> Displays hardware adjacency table information on per-module basis 55

52 Displaying Routing and Forwarding Information (Cont.) n7010# sh routing ipv /24 IP Route Table for VRF "default" /24, 1 ucast next-hops, 0 mcast next-hops *via , Ethernet9/2, [110/5], 00:02:30, ospf-1, type-1 n7010# show forwarding ipv4 route /24 module 9 IPv4 routes for table default/base Prefix Next-hop Interface / Ethernet9/2 n7010# show forwarding adjacency module 9 IPv4 adjacency information, adjacency count 1 next-hop rewrite info interface Ethernet9/2 n7010# 56 56

53 ECMP Load Sharing Up to 16 hardware load-sharing paths per prefix Use maximum-paths command in routing protocols to control number of load-sharing paths Load-sharing is per-ip flow or per-packet Use caution with per-packet load-balancing! Configure load-sharing hash options with global ip load-sharing command: Source and Destination IP addresses Source and Destination IP addresses plus L4 ports (default) Destination IP address and L4 port /16 A B Additional randomized number added to hash prevents polarization Automatically generated or user configurable value Configure per-packet load-sharing with interface ip load-sharing per-packet command Ingress interface determines if load-sharing is per-flow or per-packet! /16 via Rtr-A via Rtr-B 57

54 ECMP Prefix Entry Example n7010# sh routing ipv IP Route Table for VRF "default" /16, 2 ucast next-hops, 0 mcast next-hops *via , Ethernet9/1, [110/5], 00:03:33, ospf-1, inter *via , Ethernet9/2, [110/5], 00:00:13, ospf-1, inter n7010# sh forwarding ipv4 route module 9 IPv4 routes for table default/base Prefix Next-hop Interface / Ethernet9/ Ethernet9/2 n7010# 58

55 Identifying the ECMP Path for a Flow show routing [ipv4 ipv6] hash <sip> <dip> [<sport> <dport>] [vrf <vrf>] n7010# sh routing hash Load-share parameters used for software forwarding: load-share type: 1 Randomizing seed (network order): 0xebae8b9a Hash for VRF "default" Hashing to path * (hash: 0x29), for route: /16, 2 ucast next-hops, 0 mcast next-hops n7010# *via , Ethernet9/1, [110/5], 00:14:18, ospf-1, inter *via , Ethernet9/2, [110/5], 00:10:58, ospf-1, inter Same hash algorithm applies to both hardware and software forwarding 59

56 HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling M1 L3 Packet Flow Credit grant for fabric access 9 Supervisor Central Arbiter Return credit to pool 12 Fabric Module 1 Fabric Module 2 Fabric Module 3 VOQ arbitration and queuing 8 Transmit to fabric 10 Fabric ASIC Fabric ASIC Layer 3 6 Fabric ASIC L3 FIB/ADJ lookup Ingress and egress ACL/QoS/ NetFlow lookups Layer 3 Fabric ASIC Fabric ASIC Receive from fabric Return buffer credit 11 Submit packet headers for lookup 2nd stage ingress port QoS Receive packet from wire e1/1 VOQs Replication 10G MAC 4:1 Mux + Linksec 2 Layer 2 Forwarding LinkSec decryption 1st stage ingress port QoS Module 1 L2 ingress and egress SMAC/ DMAC lookups Return result 7 Layer 2 Forwarding Module 2 Transmit packet on wire Egress port QoS VOQs Replication 10G MAC Linksec e2/1 14 LinkSec encryption

57 Layer 3 Forwarding with F1 I/O Modules F1 modules do not natively provide Layer 3 switching Cannot inter-vlan route on their own However, one or more M1/M1-XL modules can provide proxy Layer 3 services M1 ports can proxy route for F1 modules Proxy L3 forwarding enabled by default when VDC in mixedmodule mode Packets destined to router MAC forwarded to M1 modules for Layer 3 via internal Router Port-Channel Selection of which port on which M1 module based on EtherChannel hash function Traffic requiring L3 from F1 modules traverses the fabric, vectoring toward M1 ports enabled for proxy L3 M1 module receiving such packets programmed to perform full ingress/egress L3 lookups 61

58 Proxy L3 Forwarding vlan 10 vlan 20 e2/1 e2/2 e3/1 e3/2 VLAN DMAC Dest Port 10 router_mac internal channel 2 SoC 3 4 SoC interface vlan 10 ip address /24! interface vlan 20 ip address /24 F1 Fabric ASIC Fabric Module Fabric F1 Fabric ASIC M1 Fabric Routing: DIP Next Hop Server MAC: VLAN DMAC Dest Port 20 server_2 e3/1 6 Replication VOQs Replication Forwarding 7 VOQs Replication 5 8 Replication e1/1 e1/9 e1/17 e1/25 e1/2 e1/10 e1/18 e1/26 62

59 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 64

60 IP Multicast Forwarding Forwarding tables built on control plane using multicast protocols PIM-SM, PIM-SSM, PIM-Bidir, IGMP, MLD Tables downloaded to: Forwarding engine hardware for data plane forwarding Replication engines for data plane packet replication 65

61 IP Multicast Forwarding Architecture Multicast routing processes learn routing information from neighbors/hosts IPv4 and IPv6 multicast RIBs calculate multicast routing/rp/rpf/oil information Multicast Forwarding Distribution Manager (MFDM) interfaces between MRIBs on supervisor and IP FIB on I/O modules IP FIB process programs hardware: FIB TCAM Adjacency table Multicast Expansion Table (MET) n7010# sh processes cpu egrep pim igmp PID PID Runtime(ms) Invoked usecs 1Sec Process pim igmp n7010# sh processes cpu egrep m.?rib mrib m6rib n7010# sh processes cpu egrep mfdm mfdm Supervisor PIM IGMP PIM6 ICMP6 BGP MSDP MRIB/M6RIB MFDM IP FIB IP FIB IP FIB Hardware Hardware Hardware I/O Module I/O Module I/O Module Hardware MET FIB TCAM ADJ Table module-9# sh processes cpu egrep fib ipfib module-9# 66

62 IPv4 Multicast FIB TCAM Lookup Ingress multicast packet header Compare lookup key HIT! Generate Lookup Key , , , , , , FIB TCAM Hit in FIB returns result in FIB DRAM Generate TCAM lookup key (source and group IP address) RPF, ADJ Index RPF, ADJ Index RPF, ADJ Index RPF, ADJ Index RPF, ADJ Index FIB DRAM Adj Index Identifies multicast adjacency entry Forwarding MET Index MET Index MET Index MET Index MET Index Adjacency Table Result Return lookup result Replication for each OIF in MET block Replication Replicate 67 OIFs OIFs OIFs OIFs MET MET index used to find OIFs for replication

63 Egress Replication Distributes multicast replication load among replication engines of all I/O modules with OIFs Input packets get lookup on ingress forwarding engine For OIFs on ingress module, ingress replication engine performs the replication For OIFs on other modules, ingress replication engine replicates a single copy of packet over fabric to those egress modules Each egress forwarding engine performs lookup to drive replication Replication engine on egress module performs replication for local OIFs Module 1 Fabric Module Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Replication Local OIFs MET IIF Replication Replication Local OIFs Local OIF MET MET Fabric Copy Replication Local OIFs MET 71

64 L3 Multicast Packet Flow HDR = Packet Headers DATA = Packet Data Fabric Module 1 Fabric Module 2 Fabric Module 3 VOQ queuing 10 Transmit multicast fabric distribution packet Submit packet headers for lookup 2nd stage ingress port QoS Receive packet from wire 1 Transmit to fabric Fabric ASIC Layer 3 VOQs 15 VOQs 9 Layer 2 Layer 2 delivery e1/1 11 Replication 10G MAC 4:1 Mux + Linksec Fabric ASIC Forwarding 8 6 Replicate for fabric delivery Module 1 LinkSec decryption 1st stage ingress port QoS Fabric ASIC L3 multicast FIB lookup Ingress ACL/QoS/ NetFlow lookups Egress ACL/QoS/ NetFlow lookups L2 ingress snooping lookup Return MET result L2 egress snooping lookup Layer 3 Forwarding 17 Module 2 Transmit packet on wire Egress port QoS Fabric ASIC Fabric ASIC Replication 10G MAC Linksec e2/1 Dequeue multicast distribution copy from fabric 12 Replicate for local OIF Submit packet headers for egress lookups LinkSec encryption

65 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 73

66 Classification Policies Enforce security and QoS policies based on Layer 2, Layer 3, and Layer 4 information Classification TCAM (CL TCAM) provides ACL lookups in forwarding engine 64K(non-XL) or 128K (XL) hardware entries in M1 modules 1K ingress/1k egress hardware entries per SoC in F1 modules (16K/16K entries per module) Security policies Router ACL (RACL) for IPv4, IPv6, ARP VLAN ACLs (VACLs) for IPv4, MAC Port ACLs (PACLs) for IPv4, MAC QoS classification policies for IPv4 and IPv6 Ingress (Per-L2 switchport, per-vlan, per-interface routed/subinterface) Egress (Per-VLAN, per-interface routed/subinterface) 74 74

67 Classification Architecture ACL or QoS manager receives policy via configuration Manager distributes policies to ACL/QoS Clients on I/O modules Supervisor CLI XML ACL or QoS Manager Clients perform ACL merge and program ACEs in Classification (CL) TCAM in forwarding engines ACL/QoS-C Hardware ACL/QoS-C Hardware ACL/QoS-C Hardware I/O Module I/O Module I/O Module n7010# sh processes cpu egrep "(acl qos)mgr PID" PID Runtime(ms) Invoked usecs 1Sec Process % aclmgr % ipqosmgr n7010# Hardware CL TCAM module-9# sh processes cpu egrep aclqos aclqos module-9# 75

68 CL TCAM Lookup (ACL) Packet header: SIP: DIP: Protocol: TCP SPORT: DPORT: 80 Generate Lookup Key Generate TCAM lookup key (source/dest IPs, protocol, L4 ports, etc.) SIP DIP Protocol SPORT DPORT Security ACL ip access-list example permit ip any host deny ip any host deny ip any host permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514 permit tcp any any eq 80 permit udp any any eq 161 Compare lookup key C xxxxxxx xx xx xxx xxx xxx xxx xxxxxxx xx xxx xxx xxxxxxx xx xxx xxx Forwarding Permit Deny Deny X= Mask xxxxxxx xxxxxxx 06 xxx xxxxxxx xxxxxxx 06 xxx 0017 xxxxxxx xxxxxxx 11 xxx 0202 HIT! xxxxxxx xxxxxxx 06 xxx 0050 xxxxxxx xxxxxxx 11 xxx 00A1 CL TCAM Hit in CL TCAM returns result in CL SRAM Permit Permit Permit CL SRAM Result Return lookup result Result affects final packet handling 76 Deny Deny

69 Displaying Classification Resources show hardware access-list resource utilization module <mod> n7010# sh hardware access-list resource utilization module 9 Hardware Modules Used Free Percent Utilization Tcam 0, Bank Tcam 0, Bank Tcam 1, Bank Tcam 1, Bank LOU Both LOU Operands 0 Single LOU Operands 2 TCP Flags Protocol CAM Mac Etype/Proto CAM Non L4op labels, Tcam Non L4op labels, Tcam L4 op labels, Tcam L4 op labels, Tcam n7010# 77

70 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 78

71 NetFlow NetFlow support provided by M1 I/O modules Currently no support for F1 flows NetFlow table is 512K entries, shared between ingress/egress NetFlow Hardware NetFlow entry creation CPU not involved in NetFlow entry creation/update All modules have independent NetFlow table Full and sampled NetFlow supported by hardware 79

72 NetFlow Architecture NetFlow manager receives configuration via CLI/XML NetFlow manager distributes configuration to NetFlow-Clients on I/O modules NetFlow-Clients apply policy to hardware n7010# sh processes cpu egrep nfm PID PID Runtime(ms) Invoked usecs 1Sec Process nfm module-9# sh processes cpu egrep nfp nfp module-9# Supervisor CLI XML NetFlow Manager NF-C NF-C NF-C Hardware Hardware Hardware I/O Module I/O Module I/O Module Hardware NF Table Hardware NetFlow Creation 80

73 Full vs. Sampled NetFlow NetFlow configured per-direction and per-interface Ingress and/or egress on per-interface basis Each interface can collect full or sampled flow data Full NetFlow: Accounts for every packet of every flow on interface, up to capacity of NetFlow table Sampled NetFlow: Accounts for M in N packets on interface, up to capacity of NetFlow table 83

74 Viewing NetFlow Records show hardware flow ip [detail] module <mod> n7010# sh hardware flow ip interface e9/1 module 9 D - Direction; IF - Intf/VLAN; L4 Info - Protocol:Source Port:Destination Port TCP Flags: Ack, Flush, Push, Reset, Syn, Urgent D IF SrcAddr DstAddr L4 Info PktCnt TCP Flags I 9/ :01024: A... S. I 9/ :01024: A... S. I 9/ :01024: S. < > n7010# sh hardware flow ip interface e9/1 detail module 9 D - Direction; IF - Intf/VLAN; L4 Info - Protocol:Source Port:Destination Port TCP Flags: Ack, Flush, Push, Reset, Syn, Urgent; FR - FRagment; FA - FastAging SID - Sampler/Policer ID; AP - Adjacency/RIT Pointer CRT - Creation Time; LUT - Last Used Time; NtAddr - NT Table Address D IF SrcAddr DstAddr L4 Info PktCnt TCP Flags ByteCnt TOS FR FA SID AP CRT LUT NtAddr I 9/ :01024: A... S N Y 0x000 0x x

75 NetFlow Data Export Generate NetFlow v5 or v9 export packets To NetFlow Collector I/O Module via Inband Fabric ASIC VOQs Supervisor LC CPU Aged Flows NetFlow Table Forwarding Hardware Flow Creation I/O Module Main CPU Switched EOBC LC CPU Aged Flows NetFlow Table Forwarding Hardware Flow Creation via mgmt0 Mgmt Enet To NetFlow Collector I/O Module LC CPU Aged Flows NetFlow Table Forwarding Hardware Flow Creation 87

76 Agenda Chassis Architecture Supervisor and I/O Module Architecture Forwarding Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding IP Forwarding IP Multicast Forwarding Classification Policies NetFlow Summary 89

77 Nexus 7000 Architecture Summary Variety of front-panel interface and transceiver types with LinkSec, VOQ, and other advanced hardware features I/O Modules Future-proofed chassis designs with density and airflow options Chassis Control plane protocols, system and network management Supervisor s Fabrics Forwarding s Hardware services, including unicast/multicast, bridging/routing, ACL/QoS classification, and NetFlow statistics Lossless-capable fabric with 230G/slot bandwidth to interconnect I/O modules and provide investment protection 90

78 Conclusion You should now have a thorough understanding of the Nexus 7000 switching architecture, I/O module design, packet flows, and key forwarding engine functions Any questions? 91 91

79 Q and A 93

80 BRKARC-3470 Recommended Reading 94

81 Please complete your Session Survey We value your feedback - don't forget to complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Networkers 20 th Anniversary t-shirt. All surveys can be found on our onsite portal and mobile website: You can also access our mobile site and complete your evaluation from your mobile phone: 1. Scan the Access Code (See for software, alternatively type in the access URL) 2. Login 3. Complete and Submit the evaluation 95

82