Cisco Nexus 7000 / 7700 Switch Architecture

Transcription

1

2 Cisco Nexus 7000 / 7700 Switch Architecture BRKARC-3470 Tim Stevenson Distinguished Engineer, Technical Marketing

3 Session Abstract This session presents an in-depth study of the architecture of the latest generation of Nexus 7000 and Nexus 7700 data center switches. Topics include supervisors, fabrics, I/O modules, forwarding engines, and physical design elements, as well as a discussion of key hardware-enabled features that combine to implement high-performance data center network services. BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 3

4 Session Goal To provide a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions This session will examine the Nexus 7700 system, as well as the latest additions to the Nexus 7000 This session will not examine NX-OS software architecture or other Nexus platform architectures 4 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 What Is Nexus 7000? Data-center class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7000 designed for general-purpose Data Center deployments, focused on 10G density plus 40G/100G Supervisor Engines I/O Modules Chassis Fabrics BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 What Is Nexus 7700? Data-center class Ethernet switch designed to deliver high performance, high availability, system scale, and investment protection Nexus 7700 designed for SP and MSDC Data Center deployments, focused on highdensity 40G/100G Supervisor Engine I/O Modules Chassis Fabrics BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 Nexus 7000 / Nexus 7700 Common Foundation Nexus 7000 General purpose DC switching w/10/40/100g Nexus 7700 Targeted at Dense 40G/100G deployments Common Foundation Same release vehicles, versioning, feature-sets Common configuration model Common operational model Common fabric ASICs (Fab2) and architecture Same central arbitration model Same VOQ/QoS model Identical forwarding ASICs (F2E, F3) Consistent hardware feature sets Parallel evolution of hardware capability/scale BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Nexus 7000 Chassis Family NX-OS 4.1(2) and later Nexus 7010 Nexus 7018 Back 21RU Side Side 25RU Front Front N7K-C7010 Rear Front N7K-C7018 Rear Nexus 7009 NX-OS 5.2(1) and later Nexus 7004 NX-OS 6.1(2) and later Back 14RU Side Side Side 7RU Front N7K-C7009 Rear BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 9 Front N7K-C7004 Rear

10 Nexus 7700 Chassis Family NX-OS 6.2(2) and later Nexus 7718 Back NX-OS 6.2(2) and later Nexus RU Back NX-OS 6.2(6) and later Nexus 7706 Back 14RU Front Front Front 9RU Front Rear N77-C7718 Front N77-C7710 Rear Front N77-C7706 Rear BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Key Chassis Components Nexus 7000 Common components: Supervisor engines I/O modules Power supplies (except 7004) Chassis-specific components: Fabric modules Fan trays Nexus 7700 Common components: Supervisor engines I/O modules Power supplies Chassis-specific components: Fabric modules Fan trays Common hardware components between Nexus 7000 and Nexus 7700: NONE No interchangeable hardware components between Nexus 7000 and Nexus 7700 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Supervisor Engine 2 / 2E Next generation supervisors providing control plane and management functions Supervisor Engine 2 (Nexus 7000) Supervisor Engine 2E (Nexus 7000 / Nexus 7700) Base performance High performance One quad-core 2.1GHz CPU with 12GB DRAM Two quad-core 2.1GHz CPU with 32GB DRAM Connects to fabric via 1G inband interface Interfaces with I/O modules via 1G switched EOBC N77-SUP2E Second-generation dedicated central arbiter ASIC Controls access to fabric bandwidth via dedicated arbitration path to I/O modules N7K-SUP2/N7K-SUP2E ID and Status LEDs USB Expansion Flash ID and Status LEDs Console Port Management Ethernet USB Host Ports USB Log Flash USB Expansion Flash Console Port Management Ethernet BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Nexus 7000 / 7700 I/O Module Families M1 1G and 10G M2 10G / 40G / 100G F1 10G F2 10G F2E 10G F3 40G F2E 10G F3 10G / 40G / 100G F3 closes the F/M feature gap! BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Nexus 7000 M2 I/O Modules N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L Supported in NX-OS release 6.1(1) and later 10G / 40G / 100G M2 I/O modules N7K-M224XP-23L Share common hardware architecture Two integrated forwarding engines (120Mpps) Support for XL forwarding tables (licensed) N7K-M206FQ-23L Distributed L3 multicast replication 802.1AE LinkSec on all ports Supports Nexus 2000 (FEX) connections N7K-M202CF-22L Module Port Density Optics Bandwidth M2 10G 24 x 10G (plus Nexus 2000 FEX support) SFP+ 240G M2 40G 6 x 40G (or up to 24 x 10G via breakout) QSFP+ 240G M2 100G 2 x 100G CFP 200G BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Nexus 7000 M2 I/O Module Architecture N7K-M224XP-23L / N7K-M206FQ-23L / N7K-M202CF-22L EOBC To Fabric Modules To Central Arbiters LC CPU Fabric 2 ASIC Arbitration Aggregator VOQs VOQs Forwarding Engine Forwarding Engine VOQs VOQs Replication Engine Replication Engine Replication Engine Replication Engine LinkSec + 12 X 10G MAC -or- 3 X 40G MAC -or- 1 X 100G MAC Front Panel Ports LinkSec + 12 X 10G MAC -or- 3 X 40G MAC -or- 1 X 100G MAC BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Reference: ASIC Functions M2 Modules LinkSec + MAC Provides port ASIC functions, including buffering/queuing, and performs 802.1ae encryption/decryption for front-panel ports Replication Engine Bridge between front panel port, forwarding engine, and fabric; performs multicast and SPAN replication Forwarding Engine Performs all Layer 2, Layer 3, and Layer 4 forwarding decisions and policy enforcement VOQs Interface to central arbiter and local crossbar fabric, implements Virtual Output Queuing Arbitration Aggregator Muxes arbitration requests from VOQs before sending to central arbiter on Supervisor Engine Fabric 2 Local fabric that provides first/third stage of three-stage crossbar (LC CPU Linecard CPU, runs module-specific NX-OS processes and interfaces with Supervisor Engine over EOBC) BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Nexus 7000 / 7700 F2E I/O Modules N7K-F248XP-25E / N7K-F248XT-25E / N77-F248XP-23E 7000: Supported in NX-OS release 6.1(2) and later 7700: Supported in NX-OS release 6.2(2) and later 48-port 1G/10G with SFP/SFP+ transceivers 480G full-duplex fabric connectivity System-on-chip (SoC) forwarding engine design 12 independent SoC ASICs Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) N7K-F248XP-25E N7K-F248XT-25E Interoperability with M1/M2, in Layer 2 mode on Nexus 7000 Proxy routing for inter-vlan/l3 traffic N77-F248XP-23E LinkSec support* Last 8 ports (SFP+) All 48 ports (Copper) Supports Nexus 2000 (FEX) connections * Roadmap item BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Nexus 7000 F2E Module Architecture N7K-F248XP-25E / N7K-F248XT-25E EOBC To Fabric Modules To Central Arbiters LC CPU Arbitration Aggregator Fabric 2 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC Front Panel Ports LinkSec-capable (F2E fiber) LinkSec-capable (F2E copper) BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Nexus 7700 F2E Module Architecture N77-F248XP-23E EOBC To Fabric Modules To Fabric Modules To Central Arbiters LC CPU Arbitration Aggregator Fabric 2 Fabric 2 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC 4 X 10G SoC Front Panel Ports LinkSec-capable BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Reference: ASIC Functions F2E Modules 4 X 10G SoC Four-port 10G system-on-chip; provides Port ASIC, Replication Engine, Forwarding Engine, and VOQ functions Arbitration Aggregator Muxes arbitration requests from SoCs before sending to central arbiter on Supervisor Engine Fabric 2 Local fabric that provides first/third stage of three-stage crossbar (LC CPU Linecard CPU, runs module-specific NX-OS processes and interfaces with Supervisor Engine over EOBC) BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Nexus 7000 F3 40G Module 12-port 40G QSFP+ module 480G full-duplex fabric connectivity SoC forwarding engine design 6 independent SoC ASICs Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features Fabric Services Accelerator (FSA) CPU Breakout cable support Requires Supervisor Engine 2 / 2E N7K-F312FQ-25 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Nexus 7000 F3 12-Port 40G Module Architecture EOBC To Fabric Modules To Central Arbiters FSA CPU 1G switch LC Inband x 6 x 6 to ARB x 6 Arbitration Aggregator to FSA CPU 2 X 40G SoC 1 2 X 40G SoC 2 2 X 40G SoC 3 2 X 40G SoC 4 2 X 40G SoC 5 2 X 40G SoC Front Panel Ports (QSFP+) BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Fabric Services Accelerator (FSA) High-performance module CPU with on-board acceleration engines 6Gbps inband connectivity from SOCs to FSA Multi-Mpps packet processing 2 X 2GB dedicated DRAM FSA CPU EOBC Performance/scale boost for distributed fabric services, including BFD and sampled NetFlow (roadmap) Dual-Core LC CPU Acceleration Engines Other potential applications include distributed ARP/ping processing, data plane packet analysis (wireshark), network probing, etc. 2GB DRAM I/O 2GB DRAM 6 x 1Gbps Module Inband BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Nexus 7700 F3 48-Port 1G/10G Module 48-port 1G/10G with SFP/SFP+ transceivers 480G full-duplex fabric connectivity SoC-based forwarding engine design 6 independent SoC ASICs Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features Fabric Services Accelerator (FSA) CPU LinkSec support (last 8 ports)* Supports Nexus 2000 (FEX) connections N77-F348XP-23 * Roadmap item BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 Nexus 7700 F3 48-Port 1G/10G Module Architecture EOBC To Fabric Modules To Central Arbiters FSA CPU Arbitration Aggregator 1G switch x 6 x 6 x 6 LC Inband to ARB to FSA CPU 8 X 10G SoC 1 8 X 10G SoC 2 8 X 10G SoC 3 8 X 10G SoC 4 8 X 10G SoC 5 8 X 10G SoC Front Panel Ports (SFP/SFP+) BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public LinkSec-capable 48

27 Nexus 7700 F3 40G and 100G Modules 24-port 40G QSFP+ module / 12-port 100G CPAK module 960G/1.2T full-duplex fabric connectivity SoC forwarding engine design 12 independent SoC ASICs Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QoS) and advanced features Fabric Services Accelerator (FSA) CPU 40G breakout cable support N77-F324FQ-25 N77-F312CK-26 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Nexus 7700 F3 24-Port 40G Module Architecture EOBC To Fabric Modules To Central Arbiters FSA CPU 1G switch x 6 x 12 x 12 Arbitration Aggregator LC Inband to ARB to FSA CPU 2 X 40G SoC 1 2 X 40G SoC 2 2 X 40G SoC 3 2 X 40G SoC 4 2 X 40G SoC 5 2 X 40G SoC 6 2 X 40G SoC 7 2 X 40G SoC 8 2 X 40G SoC 9 2 X 40G SoC 10 2 X 40G SoC 11 2 X 40G SoC Front Panel Ports (QSFP+) BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Nexus 7700 F3 12-Port 100G Module Architecture EOBC To Fabric Modules To Central Arbiters FSA CPU 1G switch x 6 x 12 x 12 Arbitration Aggregator LC Inband to ARB to FSA CPU 1 X 100G SoC 1 1 X 100G SoC 2 1 X 100G SoC 3 1 X 100G SoC 4 1 X 100G SoC 5 1 X 100G SoC 6 1 X 100G SoC 7 1 X 100G SoC 8 1 X 100G SoC 9 1 X 100G SoC 10 1 X 100G SoC 11 1 X 100G SoC Front Panel Ports (CPAK) BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 29 12

30 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 M-Series Forwarding Engine Hardware Two hardware forwarding engines integrated on every M2 I/O module 120Mpps (60Mpps per forwarding engine) Layer 2 bridging with hardware MAC learning 120 Mpps (60Mpps per forwarding engine) Layer 3 IPv4 60Mpps (30Mpps per forwarding engine) Layer 3 IPv6 unicast Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir) MPLS/VPLS/EoMPLS OTV RACL/VACL/PACL QoS remarking and policing policies Policy-based routing (PBR) Unicast RPF check and IP source guard IGMP snooping Ingress and egress NetFlow (full and sampled) Hardware Table M-Series Modules without Scale License M-Series Modules with Scale License MAC Address Table 128K 128K FIB TCAM 128K IPv4 / 64K IPv6 900K IPv4 / 350K IPv6 Classification TCAM (ACL/QoS) 64K 128K NetFlow Table 1M 1M BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 M-Series Forwarding Engine Architecture FE Daughter Card Egress NetFlow collection Ingress NetFlow collection L3 Engine Layer 3 FIB FIB TCAM/ ADJ Egress ACL/QoS classification Ingress ACL/QoS classification CL TCAM NetFlow Classification (ACL/QoS) Egress lookup pipeline Policing FIB TCAM and adjacency table lookups for Layer 3 forwarding ECMP hashing Multicast RPF check Ingress policing Ingress lookup pipeline Egress policing MAC Table Ingress MAC table lookups Port-channel hash result Ingress IGMP snooping lookups L2 Engine L2 Lookup (pre-l3) Ingress Parser PKT HDR From I/O Module Replication Engines L2 Lookup (post-l3) Final Results To I/O Module Replication Engines Egress MAC lookups Egress IGMP snooping lookups BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 F2E Forwarding Engine Hardware Each SoC forwarding engine services 4 front-panel 10G ports (12 SoCs per module) 60Mpps per SoC Layer 2 bridging with hardware MAC learning 60Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*) RACL/VACL/PACL QoS remarking and policing policies Policy-based routing (PBR) Unicast RPF check and IP source guard IGMP snooping Ingress sampled NetFlow FabricPath forwarding FCoE (with Sup2 / Sup2E) Roadmap on Nexus 7700 Hardware Table Per F2E SoC Per F2E Module MAC Address Table 16K 192K* FIB TCAM 32K IPv4/16K IPv6 32K IPv4/16K IPv6 Classification TCAM (ACL/QoS) 16K 192K* * Assumes specific configuration to scale SoC resources * Roadmap item BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 F3 Forwarding Engine Hardware Each SoC forwarding engine services: 8 front-panel 10G ports 2 front-panel 40G ports 1 front-panel 100G port 148Mpps per SoC Layer 2 bridging with hardware MAC learning 148Mpps per forwarding engine Layer 3 IPv4/ IPv6 unicast Layer 3 IPv4 and IPv6 multicast support (SM, SSM, Bidir*) RACL/VACL/PACL QoS remarking and policing policies Policy-based routing (PBR) Unicast RPF check and IP source guard IGMP snooping Ingress/egress* sampled NetFlow FabricPath forwarding Overlay Transport Virtualization (OTV) MPLS/VPLS/EoMPLS, LISP, VXLAN, GRE, FCoE* Hardware Table Per F3 SoC Per F3 Module MAC Address Table 64K 384K/768K** FIB TCAM 64K IPv4/32K IPv6 64K IPv4/32K IPv6 Classification TCAM (ACL/QoS) 16K 96K/192K** ** Assumes specific configuration to scale SoC resources * Roadmap items BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 F3 Forwarding Engine VOQ To/From Central Arbiter Virtual Queuing To Fabric PKT HDR Forwarding tables Ingress and egress forwarding decisions (L2/L3 lookups, ACL/QoS, features etc.) Decision Engine L2 Lookup (post-l3) From Fabric F3 SoC Ingress buffer memory Ingress Buffer MAC Table FIB/ADJ CL Layer 3 Lookups QoS / ACL L2 Lookup (pre-l3) Egress Buffer Egress fabric receive buffer PKT Ingress HDR Parser Egress Parser 1G / 10G / 40G / 100G capable interface MAC 1G / 10G / 40G / 100G 8 x 1/10G OR 2 x 40G OR 1 x 100G per ASIC PKT HDR Front-panel BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 Crossbar Switch Fabric Modules Provide interconnection of I/O modules Each installed fabric increases available per-payload slot bandwidth N77-C7718-FAB-2 N77-C7710-FAB-2 N77-C7706-FAB-2 Nexus 7000 and Nexus 7700 fabrics based on Fabric 2 ASIC Fabric Module Supported Chassis Per-fabric module bandwidth Max fabric modules Total bandwidth per slot Nexus 7000 Fabric / 7010 / Gbps per slot 5 550Gbps per slot Nexus 7700 Fabric / 7710 / Gbps per slot Tbps per slot Different I/O modules leverage different amount of available fabric bandwidth Access to fabric bandwidth controlled using QoS-aware central arbitration with VOQ N7K-C7018-FAB-2 N7K-C7010-FAB-2 N7K-C7009-FAB-2 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 Multistage Crossbar Nexus 7000 / Nexus 7700 implement 3-stage crossbar switch fabric Stages 1 and 3 on I/O modules Stage 2 on fabric modules 1 Fabric ASIC 2 Fabric 3 Fabric 4 Fabric 5 ASIC Fabric Modules ASIC ASIC Fabric ASIC 2nd stage 1 Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric Modules Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC 6 Fabric ASIC Fabric ASIC 110G (2 x 55G) 1st stage Ingress Module Egress Module 550G 3rd stage 110G (2 x 55G) 1st stage Ingress Module Nexus 7000 Nexus 7700 Egress Module BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public T 3rd stage

39 I/O Module Capacity Nexus 7000 One fabric: 440Gbps 220Gbps 330Gbps 550Gbps 110Gbps Any port can pass traffic to any other port in VDC Three fabrics: 240G M2 module has maximum bandwidth Five fabrics: per slot bandwidth 480G F2E/F3 module has maximum bandwidth Local Fabric 2 (240G) Local Fabric 2 (480G) Fabric 2 Modules Fabric 2 ASIC Fabric 2 ASIC Fabric 2 ASIC Fabric 2 ASIC Fabric 2 ASIC 5 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 What About Nexus 7004? Nexus 7004 has no fabric modules I/O modules have local fabric with 10 available fabric channels I/O modules connect back-to-back via 8 fabric channels Two fabric channels borrowed to connect supervisor engines Sup Slot 1 Fabric ASIC Fabric ASIC Sup Slot 2 2 * 55G fabric channels M2/F2E/F3 Module 3 Fabric 2 ASIC Fabric 2 ASIC M2/F2E/F3 Module 4 8 * 55G local fabric channels interconnect I/O modules (440G) BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 I/O Module Capacity Nexus 7700 One fabric: 1320Gbps 1100Gbps 220Gbps 440Gbps 660Gbps 880Gbps Any port can pass traffic to any other port in VDC Three fabrics: 480G F2E/F3 10G module has maximum bandwidth Five fabrics: 960G F3 40G module has maximum bandwidth Six fabrics: per slot bandwidth Local Fab2 #1 (480G) Local Fab2 #1 (960G) Local Fab2 #1 (1.2T) Fab2 #2 Fab2 #2 Fab2 #2 Fabric 2 Modules Fabric 2 ASICs Fabric 2 ASICs Fabric 2 ASICs Fabric 2 ASICs Fabric 2 ASICs T F3 100G module has maximum bandwidth Fabric 2 ASICs 6 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 Fabric, VOQ, and Arbitration Crossbar fabric Provides dedicated, high-bandwidth interconnects between ingress and egress I/O modules Virtual Output Queues (VOQs) Provide buffering and queuing for ingressbuffered switch architecture Central arbitration Controls scheduling of traffic into fabric based on fairness, priority, and bandwidth availability at egress ports Fabric, VOQ, and arbitration combine to provide all necessary infrastructure for packet transport inside switch BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 Buffering, Queuing, and Scheduling Buffering storing packets in memory Needed to absorb bursts, manage congestion Queuing buffering packets according to traffic class Provides dedicated buffer for packets of different priority Scheduling controlling the order of transmission of buffered packets Ensures preferential treatment for packets of higher priority and fair treatment for packets of equal priority Nexus 7000 / Nexus 7700 use queuing policies and network-qos policies to define buffering, queuing, and scheduling behavior Default queuing and network-qos policies always in effect in absence of any user configuration BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 I/O Module Buffering Models Buffering model varies by I/O module family M-series modules: hybrid model combining ingress VOQ-buffered architecture with egress port-buffered architecture F-series modules: pure ingress VOQ-buffered architecture BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Module 1 Module 2 M2 Hybrid Ingress/Egress Buffered Supervisor Engine Central Arbiter 10G M2 module used as example Fabric Module 1 Fabric Module 2 Fabric Module 3 8 ingress queues per port SP SP Local Local Local q1 q1 Ports Ports Ports Buffering / queuing / scheduling 1/2-1/12 1/13-1/23 1/14-1/24 q2 q2 Even Odd Even SP DWRR SP DWRR e2/1 q3 Ingress q3 port buffer Manages congestion of ingress forwarding/replication engines, and e2/1 e2/11 Odd VOQ 0 e1/1 - congestion e1/11 Odd toward egress VOQ destinations 1 VOQ (VQIs) 2 VOQ 3 VOQ 0 Replication Engine 0 INGRESS QUEUING POLICIES RE 1 RE 2 RE 3 Replication Engine 0 Local VQIs 2/2-2/12 Even VOQ 1 RE 1 Local VQIs 2/13-2/23 Odd VOQ 2 RE 2 Local VQIs 2/14-2/24 Even VOQ 3 RE 3 Port ASIC 0 DWRR DWRR DWRR e1/1 e1/12 Local Ports 1/13-1/24 Port ASIC 1 Port ASIC 0 e2/1 e2/12 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public e1/13-24 e2/ k DWRR SP k DWRR SP k DWRR SP Local Ports 2/13-2/24 Port ASIC 1

47 Module 1 Module 2 M2 Hybrid Ingress/Egress Buffered Supervisor Engine Central Arbiter 10G M2 module used as example Fabric Module 1 Fabric Module 2 Fabric Module 3 Fabric VOQ ASIC buffer carved by source and priority 4 priority levels SP SP Local Local Local q1 q1 Ports Ports Ports 1/2-1/12 1/13-1/23 1/14-1/24 q2 q2 Even Odd Even SP DWRR SP DWRR e2/1 q3 q3 e2/1 e2/11 Odd VOQ 0 e1/1 - e1/11 Odd VOQ 1 VOQ 2 VOQ 3 VOQ 0 Local VQIs 2/2-2/12 Even VOQ 1 Local VQIs 2/13-2/23 Odd VOQ 2 Local VQIs 2/14-2/24 Even VOQ 3 Buffering / queuing Replication Engine 0 Ingress VOQ buffer Buffers traffic and manages congestion toward egress destinations (VQIs) Port ASIC 0 DWRR DWRR DWRR FABRIC-QOS POLICY e1/1 e1/12 RE 1 RE 2 Port ASIC 1 RE 3 Local Ports 1/13-1/24 Replication Scheduling Engine 0 Egress VOQ buffer Schedules traffic toward egress destinations (VQIs) and receives frames from fabric Port ASIC 0 e2/1 e2/12 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public e1/13-24 e2/ k DWRR SP k FABRIC-QOS POLICY DWRR SP k DWRR SP RE 1 RE 2 Port ASIC 1 RE 3 Local Ports 2/13-2/24

48 Module 1 Module 2 M2 Hybrid Ingress/Egress Buffered Supervisor Engine Central Arbiter 10G M2 module used as example Fabric Module 1 Fabric Module 2 Fabric Module 3 SP SP Local Local Local Local Local Local q1 q1 Ports Ports Ports VQIs VQIs VQIs 1/2-1/12 1/13-1/23 1/14-1/24 2/2-2/12 2/13-2/23 2/14-2/24 q2 q2 Even Odd Even SP DWRR SP DWRR Buffering / Even queuing / scheduling Odd Even e2/1 q3 q3 e2/1 e2/11 Odd Egress port buffer Manages congestion at VOQ 0 e1/1 - e1/11 Odd VOQ 1 VOQ 2 VOQ 3 VOQ 0 egress physical interface VOQ 1 VOQ 2 VOQ 3 8 egress queues EGRESS QUEUING POLICIES per port Replication Engine 0 RE 1 RE 2 RE 3 Replication Engine 0 RE 1 RE 2 RE 3 Port ASIC 0 DWRR DWRR DWRR e1/1 e1/12 Local Ports 1/13-1/24 Port ASIC 1 Port ASIC 0 e2/1 e2/12 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public e1/13-24 e2/ k DWRR SP k DWRR SP k DWRR SP Local Ports 2/13-2/24 Port ASIC 1

49 F2E/F3 Ingress Buffered Nexus G F3 module used as example Supervisor Engine Central Arbiter Fabric Module 1 Fabric Module 2 Fabric Module 3 Module 1 Buffering / queuing Ingress VOQ buffer Buffers traffic and manages congestion toward egress destinations (VQIs) INGRESS QUEUING POLICY 4 ingress queues per port 8 priority levels Scheduling Egress VOQ buffer Schedules traffic toward egress destinations (VQIs) and receives frames from fabric Module 2 EGRESS QUEUING POLICY Local Ports 1/1-1/8 SOC 0 Local Ports 1/9-1/16 SOC 1 e2/1 Virtual Queuing SOC 2 SP q1 q2 q3 e1/17 e1/24 Ingress Buffer Local Ports 1/25-1/32 SOC 3 Local Ports 1/33-1/40 SOC 4 Local Ports 1/41-1/48 SOC 5 e1/17 e1/24 e1/1-8 e1/9-16 e1/25-32 e1/33-40 e1/41-48 SP q1 q2 q3 SP SOC 0 DWRR SP DWRR e2/1 e2/8 Egress Buffer e2/1 e2/8 Local VQIs 2/9 2/16 SOC 1 SOC 1 SOC 1 SOC 1 SOC 1 e2/9-48 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 FAQ: What Is a VQI? VQI = Virtual Queuing Index A Destination Across the Fabric For M2 / F2E / F3 10G modules, VQI == 10G interface For F3 40/100G modules, VQI == 40/100G interface For M2 40/100G ports, uses multiple 10G VQIs BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 M2 Module 40G and 100G Flow Limits Internal to Nexus 7000 System On the Wire (40G) Ingress Modules Spines Spines Spines Spines Fabrics Destination VQIs 10G 10G 40G 40G 100G 1 packet n bits 40G Port Tx 1 64/66B Encoding Tx 2 Tx 3 Tx VQI 1 VQI 4 VQIs 4 VQIs 10 VQIs 66 bits Egress Interfaces Each Virtual Queuing Index (VQI) sustains 10G traffic flow All packets in given 5-tuple flow hash to single VQI Packets split into 66-bit code words Four code words transmitted in parallel, one on each physical Tx fiber Single-flow limit is 10G No per-flow limit imposed splitting occurs at physical layer BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 F3 Module 40G and 100G Flow Limits Internal to Nexus 7000 / 7700 System Ingress Modules Spines Spines Spines Spines Fabrics Destination VQIs 10G 10G 40G 40G 100G 1 VQI 1 VQI 1 VQI 1 VQI 1 VQI Egress Interfaces Virtual Queuing Index (VQI) sustains 10G, 40G, or 100G traffic flow based on destination interface type No single-flow limit full 40G/100G flow support BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 Hardware Layer 2 Forwarding Process Layer 2 forwarding traffic steering based on destination MAC address MAC table lookup drives Layer 2 forwarding Source MAC and destination MAC lookups performed for each frame, based on {VLAN,MAC} pairs Source MAC lookup drives new learns and refreshes aging timers Destination MAC lookup dictates outgoing switchport BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 M2 L2 Packet Flow HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling Credit grant for fabric access Supervisor Engine Central Arbiter Return credit to pool Fabric Module 1 Fabric Module 2 Fabric Module 3 VOQ arbitration and queuing Receive from fabric Return buffer credit Module 1 Module 2 ACL/QoS/ NetFlow lookups L2 SMAC/ DMAC lookups Port-channel hash result Forwarding Engine Layer 3 Engine Fabric 2 ASIC VOQs VOQs Round-robin transmit to fabric Hash-based uplink and VQI selection Round-robin transmit to VQI Forwarding Engine Layer 3 Engine Fabric 2 ASIC VOQs VOQs Return result destination + hash result Static or hashbased RE uplink selection Ingress port QoS Layer 2 Engine Replication Engine Replication Engine 10G/40G/100G MAC / LinkSec Submit packet headers for lookup LinkSec decryption Egress port QoS Layer 2 Engine Replication Engine Replication Engine 10G/40G/100G MAC / LinkSec Static downlink selection LinkSec encryption Receive Transmit packet from e1/1 packet on e2/2 wire wire BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 F2E / F3 L2 Packet Flow HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling Credit grant for fabric access Supervisor Engine Central Arbiter Return credit to pool Fabric Module 1 Fabric Module 2 Fabric Module 3 Fabric Module 4 Fabric Module 5 Transmit to fabric VOQ arbitration Submit packet headers for lookup Receive from fabric Return result destination Ingress port QoS (VOQ) VOQ e1/1 DE SoC Module 1 Receive packet from wire Ingress L2 SMAC/ DMAC lookups, ACL/QoS lookups, NetFlow sampling Transmit packet on wire e2/2 VOQ SoC Module 2 Egress port QoS (Scheduling) Return buffer credit BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 Layer 3 Forwarding Nexus 7000 decouples control plane and data plane Forwarding tables built on control plane using routing protocols or static configuration OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing Tables downloaded to forwarding engine hardware for data plane forwarding FIB TCAM contains IP prefixes Adjacency table contains next-hop information BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 Hardware Layer 3 Forwarding Process FIB TCAM lookup based on longest-match destination prefix comparison FIB hit returns adjacency, adjacency contains rewrite information (next-hop) Pipelined forwarding engine architecture also performs ACL, QoS, and NetFlow lookups, affecting final forwarding result BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 IP FIB TCAM Lookup HIT! Generate Lookup Key xx xx xx xx xx xx xx xx FIB TCAM Generate TCAM lookup key (destination IP address) Compare lookup key Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Index, # next-hops Hit in FIB returns result in FIB DRAM Index, # next-hops FIB DRAM Ingress unicast IP packet header Load-Sharing Hash # nexthops Adj Index Flow Data Adjacency index identifies ADJ block to use Modulo function selects exact next hop entry to use Forwarding Engine Next-hop 1 (IF, MAC) Next-hop 2 (IF, MAC) Next-hop 3 (IF, MAC) Next-hop 4 (IF, MAC) Next-hop 5 (IF, MAC) Next-hop 6 (IF, MAC) Next-hop 7 (IF, MAC) Adjacency Table BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 60 mod Offset Return lookup result Result

61 M2 L3 Packet Flow HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling Credit grant for fabric access Supervisor Engine Central Arbiter Return credit to pool Fabric Module 1 Fabric Module 2 Fabric Module 3 VOQ arbitration and queuing Receive from fabric Return buffer credit L3 FIB/ADJ lookup Ingress and egress ACL/QoS/NetFlow lookups L2 ingress and egress SMAC/ DMAC lookups Port-channel hash result Module 1 Forwarding Engine Layer 3 Engine Fabric 2 ASIC VOQs VOQs Round-robin transmit to fabric Hash-based uplink (and VQI) selection Round-robin transmit to VOQ Forwarding Engine Layer 3 Engine Module 2 Fabric 2 ASIC VOQs VOQs Return result destination + hash result Static or Hash-based uplink selection Ingress port QoS Layer 2 Engine Replication Engine Replication Engine 10G/40G/100G MAC / LinkSec Submit packet headers for lookup LinkSec decryption Egress port QoS Layer 2 Engine Replication Engine Replication Engine 10G/40G/100G MAC / LinkSec Static RE downlink selection LinkSec encryption Receive Transmit packet from e1/1 packet on e2/2 wire wire BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 HDR = Packet Headers DATA = Packet Data CTRL = Internal Signaling F2E / F3 L3 Packet Flow Credit grant for fabric access Supervisor Engine Central Arbiter Return credit to pool Fabric Module 1 Fabric Module 2 Fabric Module 3 Fabric Module 4 Fabric Module 5 Transmit to fabric Receive from fabric VOQ arbitration Submit packet headers for lookup Return result destination VOQ DE SoC Module 1 L2 ingress and egress SMAC/ DMAC lookups L3 FIB/ADJ lookup Ingress and egress ACL/QoS lookups, NetFlow sampling VOQ SoC Module 2 Ingress port QoS (VOQ) e1/1 Receive packet from wire Transmit packet on wire e2/2 Egress port QoS (Scheduling) Return buffer credit BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 Layer 3 Forwarding Module Interoperability Models Two interoperability models for L3 forwarding: Proxy Forwarding Ingress Forwarding with Lowest Common Denominator BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 Proxy Forwarding Model Conceptual From F2E perspective, Router MAC reachable through giant port-channel All packets destined to Router MAC forwarded through fabric toward one member port in that channel vlan 10 All F2E modules Up to 128 links All M2 modules interface vlan 10 ip address /24! interface vlan 20 ip address / vlan 20 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 Proxy Forwarding Model Actual Can be up to 128 M2 VQIs VLAN DMAC e1/ vlan 10 FE SoC Dest Port 10 router_mac internal_channel (e3/1-24,e4/1-24) EtherChannel Hash Function hash_input (from packet) select_member_port 1 Mod 1 4 Fabric F2E 2 Fabric 3 M2 Fabric Mod 3 Programming of all F2E forwarding engines Replication Engine 6 VOQs Replication 5 Engine FE 9 e3/13 7 FE VOQs Replication Engine Replication Engine e3/20 e3/19 e3/14 e3/8 e3/7 e3/2 e3/1 8 Ingress MAC: VLAN DMAC Dest Port 10 router_mac L3_lookup Routing: DIP Next Hop server_2_mac (v20) Egress MAC: VLAN DMAC 20 server_2_mac e2/1 Dest Port 10 e2/ vlan 20 FE SoC Mod 2 Fabric F2E Fabric Fabric Modules M2 Fabric Mod 4 VOQs FE FE VOQs Replication Engine Replication Engine Replication Engine Replication Engine BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 65 e4/20 e4/19 e4/14 e4/13 e4/8 e4/7 e4/2 e4/1 Programming of all M2 forwarding engines interface vlan 10 ip address /24! interface vlan 20 ip address /24

66 Proxy Layer 3 Forwarding Process 1. Host connected to F2E module (e1/1) in VLAN 10 wants to send traffic to host connected to F2E module (e2/1) in VLAN 20 host sends traffic to DMAC of VLAN 10 SVI (after ARPing for gateway) 2. SoC ASIC on F module does DMAC lookup for Router MAC, output port for such traffic is internal L3 port-channel comprised of all the M1/M2 front-panel ports (i.e., router can be reached out any of those ports) 3. Packet hashes to one of the available links (in this case it hashes to e3/6) 4. Packet is sent across the fabric toward e3/6; Replication engine owning e3/6 receives packet from fabric, but packet is treated as if it was received from the wire 5. Replication engine sends packet headers to M forwarding engine 6. Ingress MAC lookup hits Router MAC entry, triggering full L3 lookup 7. Dest IP lookup returns next hop of (Server 2 is directly connected) with DMAC server_2_mac 8. Egress MAC lookup hits server_2_mac entry, returning egress port where Server 2 is connected (e2/1) 9. Forwarding engine returns result to replication engine, indicating output VLAN, rewrite MAC addresses, and output port (e2/1) 10. Packet is sent back across the fabric to module 2, which forwards it out e2/1 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 Ingress Forwarding with Lowest Common Denominator Model F3 module interoperability always Ingress Forwarding NO proxy forwarding with F3 The ingress module makes all the forwarding decisions Supported feature set based on Lowest Common Denominator Feature available if all modules support the feature Not all features supported by software today VDC Type Layer 2 Layer 3 vpc Fabric Path VXLAN FEX MPLS OTV LISP FCoE Table Sizes F3 F3 size M2 + F3 F3 size F2/F2E + F3 F2E size BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Interoperability Forwarding Model Matrix Module Combination M1 + M2 M + F1 M + F2E F2 + F2E F2 + F2E + F3 M2 + F3 M2 + F2/F2E + F3 Interoperability Model Lowest Common Denominator Proxy Forwarding Proxy Forwarding Lowest Common Denominator Lowest Common Denominator Lowest Common Denominator NOT SUPPORTED BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 What Is Classification? Matching packets Layer 2, Layer 3, and/or Layer 4 information Used to decide whether to apply a particular policy to a packet Enforce security, QoS, or other policies Some examples: Match TCP/UDP source/destination port numbers to enforce security policy Match destination IP addresses to apply policy-based routing (PBR) Match 5-tuple to apply marking policy Match protocol-type to apply Control Plane Policing (CoPP) etc. BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 CL TCAM Lookup ACL Packet header: SIP: DIP: Protocol: TCP SPORT: DPORT: 80 Generate Lookup Key Generate TCAM lookup key SIP DIP Pr SP DP Security ACL ip access-list example permit ip any host deny ip any host deny ip any host permit tcp any any eq 22 deny tcp any any eq 23 deny udp any any eq 514 permit tcp any any eq 80 permit udp any any eq 161 Compare lookup key to CL TCAM entries tcp xxxxxxx xx xx xxx xxx xxx xxxxxxx xx xxx xxx Forwarding Engine Permit Deny Results Comparisons (X = Mask ) xxxxxxx xx xxx xxx xxxxxxx xxxxxxx tcp xxx xxx xxxxxxx xxxxxxx tcp xxx 23 Deny Permit Deny xxxxxxx xxxxxxx udp xxx 514 HIT! xxxxxxx xxxxxxx tcp xxx 80 xxxxxxx xxxxxxx udp xxx 161 SIP DIP Pr SP DP CL TCAM Hit in CL TCAM returns result in CL SRAM Deny Permit Permit CL SRAM Result Return lookup result Result affects final packet handling BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 CL TCAM Lookup QoS Packet header: SIP: DIP: Protocol: TCP SPORT: DPORT: 80 Compare lookup key Comparisons (X = Mask ) Generate Lookup Key tcp xxxxxxx xx xx xx xxx xxx xxxxxxx xx xx xxx xxx xx xxxxxxx udp tcp xxx xxx HIT! xx xxxxxxx tcp xxx xxx xxxxxxx xx tcp xxx 23 CL TCAM Generate TCAM lookup key SIP DIP Pr SP DP SIP DIP Pr SP DP Hit in CL TCAM returns result in CL SRAM Forwarding Engine Policer ID 1 Policer ID 1 Remark DSCP 32 Remark DSCP 40 Remark IP Prec 3 CL SRAM QoS Classification ACLs ip access-list police permit ip any /24 permit ip any /24 ip access-list remark-dscp-32 permit udp /24 any ip access-list remark-dscp-40 permit tcp /24 any ip access-list remark-prec-3 permit tcp any /24 eq 23 Result Return lookup result Results Result affects final packet handling BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 NetFlow NetFlow collects flow data for packets traversing the switch Each module maintains independent NetFlow table M2 F2E / F3 Per-interface NetFlow Yes Yes NetFlow direction Ingress/Egress Ingress only Full NetFlow Yes No Sampled NetFlow Yes Yes FSA Assist for Sampled NetFlow No F3 only (future) Bridged NetFlow Yes Yes Hardware Cache Yes No Software Cache No Yes Hardware Cache Size 512K entries per forwarding engine BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 74 N/A NDE (v5/v9) Yes Yes

75 Full vs. Sampled NetFlow NetFlow collects full or sampled flow data Full NetFlow: Accounts for every packet of every flow on interface Available on M-Series modules only Flow data collection up to capacity of hardware NetFlow table Sampled NetFlow: Accounts for M in N packets on interface Available on both M2 (ingress/egress) and F2E/F3 (ingress only) M2: Flow data collection up to capacity of hardware NetFlow table F2E/F3: Flow data collection for up to ~1000/3000pps per module F3 (future): Increased per-module sampling rate leveraging on-board Fabric Services Accelerator (FSA) complex BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 Sampled NetFlow Details Random packet-based sampling M:N sampling: Out of N consecutive packets, select M consecutive packets and account only for those flows On M2, sampled packets create hardware NetFlow table entry On F2E/F3, sampled packets sent to LC CPU via module inband Rate limited to ~1000pps per module Software multiplies configured sampler rate by 100 on F2E/F3 modules Example: when using 1 out-of 100 sampler on F2/F2E interface, sampled rate becomes 1:10000 BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 NetFlow on M2 Modules To NetFlow Collector Generate NetFlow v5 or v9 export packets M2 Module via Supervisor Inband Fabric ASIC VOQs Supervisor Engine LC CPU Aged Flow Info Forwarding Engine M2 Module NetFlow Table Hardware Flow Creation Main CPU Switched EOBC LC CPU Aged Flow Info Forwarding Engine NetFlow Table Hardware Flow Creation via mgmt0 Mgmt Enet To NetFlow Collector M2 Module Forwarding Engine BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 77 LC CPU Aged Flow Info NetFlow Table Hardware Flow Creation

78 Sampled NetFlow on F2E/F3 Modules via Supervisor Inband To NetFlow Collector Fabric ASIC Age flows and generate NetFlow v5 or v9 export packets Supervisor Engine Aged Flows DRAM NetFlow Cache FSA CPU via Module Inband Populate cache based on received samples Sampled Packets F3 Module SoC Decision Engine Data Flow VOQs Main CPU Switched EOBC Aged Flows DRAM NetFlow Cache FSA CPU via Module Inband Sampled Packets F3 Module SoC Decision Engine Data Flow via mgmt0 Mgmt Enet To NetFlow Collector Aged Flows DRAM NetFlow Cache LC CPU via Module Inband Sampled Packets F2E Module SoC Decision Engine Data Flow BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 Agenda Chassis Architecture Supervisor Engine and I/O Module Architecture Forwarding Engine Architecture Fabric Architecture I/O Module Queuing Layer 2 Forwarding Layer 3 Forwarding Classification NetFlow Conclusion BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 Nexus 7000 / Nexus 7700 Architecture Summary I/O Modules Supervisor Engines Chassis Fabrics BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 Conclusion You should now have a thorough understanding of the Nexus 7000 / Nexus 7700 switching architecture, I/O module design, packet flows, and key forwarding engine functions Any questions? BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 Reference: Acronym Decoder ACL Access Control List ADJ Adjacency ASIC Application Specific Integrated Circuit CFP C Formfactor Pluggable CoPP Control Plane Policing COS Class of Service DSCP Differentiated Services Code Point DWRR Deficit Weighted Round Robin ECMP Equal Cost Multi Path EEE Energy Efficient Ethernet EOBC Ethernet Out-of-Band Channel FCoE Fiber Channel over Ethernet FE Forwarding Engine FEX Fabric Extender (Nexus 2000 family) FIB Forwarding Information Base FRU Field Replaceable Unit GRE Generic Route Encapsulation HSRP Hot Standby Router Protocol IGMP Internet Group Management Protocol MPLS Multiprotocol Label Switching NDE NetFlow Data Export OTV Overlay Transport Virtualization PACL Port ACL PBR Policy-Based Routing PIM Protocol Independent Multicast QoS Quality of Service QSFP+ 40G Quad Small-Formfactor Pluggable RACL Router ACL RE Replication Engine RPF Reverse Path Forwarding RU Rack Unit SFP+ 10G Small-Formfactor Pluggable SoC System-on-chip/switch-on-chip TCAM Ternary CAM urpf Unicast RPF VACL VLAN ACL VDC Virtual Device Context VOQ Virtual Output Queuing VQI Virtual Queuing Index XL Refers to forwarding engine with larger FIB and ACL TCAMs BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 83 83

84 Continue Your Education Demos in the Cisco Campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings BRKARC Cisco and/or its affiliates. All rights reserved. Cisco Public 84 84

85

86