Cisco Nexus 6001/6004 Architecture

Transcription

1

2 Cisco Nexus 6001/6004 Architecture Eddie Tan Principal Engineer-Technical Marketing

3 Agenda Nexus 6004/6001 Overview Nexus 6004/6001 Internal Architecture Nexus 6004/6001 Forwarding Lookup Nexus 6004/6001 Transceivers/Cables Nexus 6004/6001 Access Control List Nexus 6004/6001 SPAN Nexus 6004/6001 Multicast Nexus 6004/6001 QoS Nexus 6004/6001 Switch Fabric Mode 3

4 Nexus 6004/6001 Overview

5 Introducing Cisco Nexus 6004 High Performance High Scalability Feature-Rich 96 x 40GE Line rate L2 and L3 L2 and L3 features 1-microsecond 384x10GE latency with all Up to 256,000 MAC FEXlink frame sizes Up to 128,000 ARP vpc FabricPath 40-Gbps flow 32,000 LPM TRILL 40-Gbps FCoE 16,000 bridge FabricPath with 25-MB buffer per 3x domains QSFP segment ID* 31 SPAN sessions (16 at FCS) * Some features in this slide are not supported by software at FCS Adapter-FEX/VM- FEX Visibility and Analytics Line-rate SPAN Sampled NetFlow* Micro-burst and buffer monitoring* Latency monitoring* Conditional SPAN- SPAN on drop- SPAN on higher latency* 5

6 Cisco Nexus 6004 Chassis Port-Side View Chassis depth: 30 in. Chassis width: 17.5 in. 48 Fixed QSFP Interfaces 12 QSFP ports LEM Linecard Expansion Module 4RU 30 in. 6

7 Cisco Nexus 6004 Chassis Power Supply and Fans Six power supply slots; a minimum of three is required. Support both and redundancy. Each power supply is rated 1100W, VAC. The chassis has four fan trays. A minimum of three is required. Power Supply Grid Redundancy or Redundancy Console Mgmt0 USB Fan Module 3+1 Redundancy 7

8 Cisco Nexus 6004 Airflow Front-to-back or back-to-front airflow Port-side exhaust at FCS(First Customer Shipment) Support port-side intake (reversed airflow) from 6.0(2)N1(2) Different power supply and fan modules required for different airflow directions Airflow direction at FCS 8

9 Nexus 6001 Chassis Port-Side View Built with same ASIC family as Nexus 6004 Chassis depth: 30 in. Chassis width: 17.3 in. 1RU 48 fixed 1/10G interfaces 4 Fixed QSFP interface 9

10 Nexus 6001 Airflow Front to back or back to front air flow Different PS and Fan modules are required for different air flow directions Front to back air flow 10

11 Twelve QSFP Line-Card Expansion Modules Provide 12 QSFP interfaces Support 40, 10GE, FCoE Offer same performance, features, and scalability as fixed ports Support OIR 11

12 Unified Port Line-Card Expansion Module For Nexus 6004 Planned 1HCY14 16 SFP+ unified ports expansion module 2/4/8/FC port 1G/10GE and FCoE For Nexus 6004 only SFP+ port allows support for more variety of optical transceiver

13 Cisco Nexus 6001 Control Plane Elements CPU DRAM Boot/BIOS flash memory Four core 2-GHz Intel Gladden 8 GB of DDR3 in four DIMM slots 8 GB to store upgradable and golden version of (BIOS + bootloader) image Onboard fault log (OBFL) 64 MB of flash memory to store hardware-related fault and reset reason NVRAM Management Interfaces 6 MB of SRAM to store syslog and licensing information RS-232 console port 10/100/1000BASE-T management port USB port 13

14 Cisco Nexus 6004 Control Plane Elements CPU DRAM Program Store Four core 2-GHz Intel Gladden 16 GB of DDR3 in four DIMM slots 80 GB of isata flash for base system storage and partitioned to store image, configuration, log Boot/BIOS flash memory 8 GB to store upgradable and golden version of (BIOS + bootloader) image Onboard fault log (OBFL) NVRAM Management Interfaces 64 MB of flash memory to store hardware-related fault and reset reason 6 MB of SRAM to store syslog and licensing information RS-232 console port 10/100/1000BASE-T management port USB port 14

15 Cisco Nexus 6001 Power Supply Characteristics Power consumption is measured with fully populated system with optical module. AC Power Supply Properties Typical operating power N6K-C W Maximum power 1100 W Input voltage Frequency 90 to 240 VAC 47 to 63 HZ Efficiency 95 to 98% (50 to 100%) Hot-swappable Yes RoHS Compliance Yes 15

16 Cisco Nexus 6004 Power Supply Characteristics Each AC PSU is rated as 1100W; a minimum of 3 PSUs is required. Power consumption is measured with fully populated system with optical module. AC Power Supply Properties Typical operating power Maximum power Input voltage Frequency N6K-C Q 2800 W 3300 W 90 to 240 VAC 50 to 60 HZ Efficiency 95 to 98% (50 to 100%) Hot-swappable Yes RoHS Compliance Yes 16

17 N6004 and N5500 Comparison The table list the Nexus 6004 hardware features and scalability numbers. Please check software roadmap for availability Feature N6004 N5548/N5596 (Trident+) Latency ~1 us 1.8us L2 Throughput Line rate Line rate L3 Throughput Line rate 160Gbps Native 40GE support Yes No (4xQSFP module on the roadmap) 1GE support Future hardware Yes Unified Port Future hardware Yes FEX Scaling with L2, 16 with L3 MAC Table 256K(shared with ARP/ND) 32K IPv4 Unicast routes 32K 16K IPv4 ARP/host route entries 256K(shared with MAC) 16K(shared with IPv6 ND and mroute) IPv4 multicast routes 32K 8K(shared with ARP) 17

18 N6k and N5500 Comparison (cont.) The table list the Nexus 6004 hardware features and scalability numbers. Please check software roadmap for availability Feature N6004 N5548/N5596 (Trident+) IPv6 unicast routes 8K(shared with IPv4) 8k(shared with IPv4) Ipv6 ND entry/host route entries 85K(shared with MAC and ARP) 8K(shared with mroute and ARP) Ipv6 multicast routes 32K VLAN 4K 4K Bridge Domain 16K 4K VRF 4K 1K Segment ID Yes No ACL 4K 4K 8K(shared with ARP, ND. Limit is with /64 routes) L4ops for ACL 24 total(16 for UDP/TCP, 8 for TCP flag) 8 for UDP and 8 for TCP Packet Buffer 25MB per 3xQSFP(or 12x10G) 640KB per 10G port Buffer management Dedicated plus shared Dedicated per port 18

19 N6k and N5500 Comparison (cont.) The table list the Nexus 6004 hardware features and scalability numbers. Please check software roadmap for availability Feature N6004 N5548/N5596 (Trident+) SPAN 31 bi-directional (16 at FCS) 4 bi-directional ERSPAN source session 16 4 ERSPAN destination session 16 No Line rate SPAN Yes No Prioritize data over SPAN Yes(through scheduler) Yes(SPAN policing) Conditional SPAN Yes No Multicast Replication Fabric and egress replication Fabric PIM-BiDir Yes No IP based forwarding for IGMP snooping Yes No 19

20 24xFEX FEX Scalability 24 FEX per Cisco Nexus 6004 in L2 and L3 at FCS(48x FEX with N6004 in L2 planned for 6.0(2)N2(1) release). Support for all different types of FEX including HP FEX and Fujitsu FEX for blade servers (except first model Cisco Nexus 2148) Flexible connectivity: rack mount, blade, 1G/10G, 10GBase-T, LACP NIC teaming Cisco Nexus 6000 Cisco Nexus 2000 FEX + 20

21 24 FEX 24 FEX 24 FEX 24 FEX 24 FEX 24 FEX 48 FEX 48 FEX 24 FEX FEX Topology and Scalability Reference Without vpc/vpc+ With vpc/vpc+ 21

22 Nexus 6000 Internal Architecture

23 Cisco Nexus 6001 Internal Architecture Switch Fabric Unified Fabric Controller(UFC) UPC 1 UPC 2 UPC 3 UPC 4 UPC 5 UPC 6 Supervisor UPC-0 48xSFP+ 4xQSFP+ CPU 23

24 Cisco Nexus 6004 Internal Architecture Switch Fabric UFC-1 UFC-2 UFC-3 UFC-4 Supervisor UPC 1 UPC 2 UPC 16 UPC 17 UPC 32 UPC-0 48xQSFP (Fixed interfaces) 48xQSFP (with 4 expansion modules) CPU 24

25 Unified Port Controller - UPC ASIC Multimode MAC; built-in PHY for 1, 10, and 40 GE Packet parsing and rewriting Lookup engine and access control: L2, L3, FabricPath, TRILL, ACL, FCoE, and policing Buffering and queuing: Buffer management, PFC for lossless traffic, queuing (Strict Priority Queuing and DWRR), and packet replication (SPAN and multicast) Extra fabric bandwidth for SPAN and multi-destination traffic Front Panel Ports Twelve 10 GE UPC Fabric Connections 224Gbps 448Gbps Three 40 GE UPC 224Gbps 448Gbps 25

26 Unified Crossbar Fabric - Fabric ASIC Unified Crossbar Fabric: Can operate at 10 GE optimized mode or 40 GE optimized mode Lossless Fabric In 10G mode: 192x384 crossbar. Up to 192 connections to ingress UPC. Up to 384 connection to egress UPC 14Gbps per link In 40G Mode: 48x96 crossbar Up to 48 connections to ingress UPC. Up to 96 connections to egress UPC 56Gbps per link 3 x 40GE 3 x 40GE Unified Port Controller 1 Unified Port Controller 32 Fabric 224 G (4 x 56) Fabric 448 G (8 x 56) 224 G (4 x 56) 448 G (8 x 56) 4 UCF (40G) 26

27 Cisco Nexus 6000 Packet Processing Flow Ingress Pipeline 1, 10, and 40 GE MAC Parser Security ACL QoS ACL SPAN Filter Policing/Stats Storm Control VLAN Table MAC Table ARP/ND Table IPv4/Ipv6 RoutingTable FabricPath/TRILL FC routing Table Forwarding Lookup ACL Ingress Policing Ingress SPAN Replication Packet Rewrite Ingress UPC Buffer Allocation Buffer Accounting Unicast VOQ PAUSE Signal (8 per egress port) Buffer Management Buffer 8,000 Multicast VOQ Unified Crossbar Fabric 27

28 Unified Crossbar Fabric Cisco Nexus 6000 Packet Processing Flow Egress Pipeline Egress UPC unicast buffer Multicast congestion management Unicast Queue Buffer Management Scheduling Multicast Replication Egress ACL Egress Policing* Editing MAC Buffer Multicast Queue TX SPAN Replication To Ingress Buffer Management *Egress policing is not supported with current software 28

29 Nexus 6000 Forwarding

30 UPC Forwarding Lookup Virtual Interface Table (8192) Host Table(MAC, ARP/ND, (*,G) (256K) LPM Table (32K) FabricPath Table (8K) Mroute Table (64K) ACL Search Engine (4K) Fibre Channel Multipath Table Collect Interface and VLAN State Determine Destination (ingress only) Ethernet Learning Policy Enforcement Multipath Expansion Vlan Translation Table (32K) Vlan State Table (16K) Fibre Channel Switch Table (4K) Multicast Vector Table (8K) RBACL Label Table (4K) Binding Table (4K) Zoning Table (4K) PortChannel Table 30

31 UPC Forwarding Lookup Collect Interface and VLAN Configuration Identify the source interface FEX HIF port as N6k logical interface VM-FEX/Adapter-FEX Identify the incoming VLAN. Map VLAN to internal BD(Bridge Domain) Map dot1q VLAN ID to internal BD Map incoming double dot1q (24-bit) ID to internal BD FabricPath Segment ID, VLAN translation, QinQ(Not supported in current software) N6k hardware support 16K BD Logical Interface Table (8192) Collect Interface and VLAN State Vlan Translation Table (32K) Vlan State Table (16K) Check if the port is in forwarding or blocking 31

32 UPC Forwarding Lookup Switching or Routing My_Router_MAC table: 512-entry MAC table. When the destination MAC of incoming packets matches MAC in the table then the packet will be L3 forwarded. MY_Router_MAC(512-entry) VLAN Index MAC Routing VLAN Destination MAC bb.a4c c07.ac01 Hit? No Switching Yes Egress port from L3 table 512 Egress port from MAC table 32

33 FHRP Group Scaling interface Vlan100 no shutdown no ip redirects ip address /24 hsrp 1 ip interface Vlan101 no shutdown no ip redirects ip address /24 hsrp 1 ip Traditionally FHRP(HSRP or VRRP) group ID is mapped to VLAN ID for operational consistency In vpc enabled design both node is actively forwarding data Nexus 6000 has maximum 500 FHRP unique group ID support Does not mean one can not use more then 500 VLANs or HSRP group Use same FHRP group ID across SVI for better FHRP group scalability Same FHRP group ID means same FHRP VMAC. All VLANs can share same entry in the My_Router_MAC table. All L3 interfaces on N6000 use same MAC address 33

34 256,000 Entries Cisco Nexus 6000 Key Forwarding Tables Host table: 256,000 -entry hashing table; actual capacity is slightly less than 256,000 Host table: Shared between MAC, ARP, and ND and /32 host route Host table FCS carving: 128,000 MAC, 128,000 IP host LPM table: 32,000 entries. Also known as summary routes Host Table* LPM Table(32,000) Mroute table: 64,000 Mroute entries Table(64,000)* MAC Region Summary Routes (S,G) IP Host Region (ARP/ND/Host route/(*,g)) * Hardware table size. Please check configuration limit for software scaling 34

35 Check N6K Unicast Forwarding Table Usage N6kL3a# sh ip route sum IP Route Table for VRF "default" Total number of routes: Total number of paths: Best paths per protocol: Backup paths per protocol: am : am : 3 local : 19 direct : 19 static : 2 broadcast : 41 hsrp : 18 ospf-1 : Number of routes per mask-length: /8 : 3 /22: 1 /24: /32: N6kL3a# sh hardware profile status Max Mcast Routes = Used Mcast Routes = 4. Total LPM Entries = Used Unicast IPv4 LPM Entries = Used Unicast IPv6 LPM Entries = 3. Used Unicast Host4 Entries in LPM= 0. Used Unicast Host6 Entries in LPM= 0. Used Multicast IPv4 LPM Entries = 24. Reserved LPM Entries for future use = 2. Used Host Entries = Used Unicast Host Entries in Host (Total) = Used Unicast Host4 Entries in Host = Used Unicast Host6 Entries in Host = 0. Used Multicast Host Entries in Host (Total) = 0. Used Multicast Host4 Entries in Host = 0. Used Multicast Host6 Entries in Host = 0. 35

36 Host Table LPM Table Cisco Nexus 6000 Unicast Table Scaling Each IPv6 ND(Neighbor Discovery) entry consumes two entries in host table. Each IPv6 route consumes four entries in LPM table. Upcoming release-6.0(2)n2(1) provides CLI to adjust size of each region in Host Table / ::/64 MAC: 00:02:B3:01:02:03 IPv4: IPv6: 2001::0DB8:800:200C:417A MAC Region 00:02:B3:01:02:03 IP Host Region :0:0:0: 0DB8:800:200C:417A ARP :0 0:0 0:0 0:0 Two Hardware Entries for IPv6 ND IPv4 Route Four Hardware Entries for IPv6 Route 36

37 Cisco Nexus 6000 Host-Table Scaling Host table is hashing table. Actual capacity will be slightly lower than the number in the table. The number reflects the hardware capacity and is not necessarily the software scaling number at FCS. Flexible table carving (roadmap feature after Cisco IOS Software Release 6.0(2)N1(1)) allows maximum scaling in different deployment scenario. Deploy Scenario L2 switch L2L3 gateway with IPv4 only* L2L3 gateway with IPv6 only* L2L3 gateway with dual stack* Scalability 256,000 MAC 128,000 virtual machines 85,000 virtual machines 64,000 virtual machines * Assume one IPv4 or IPv6 per VM. Hardware scaling number. 37

38 Unicast L3 Forwarding Lookup Identify Egress Port Host Table (IP Host Region) 64-way ECMP for OSPF/EIGRP/BGP routes Support ECMP for host routes ARP/ND/Host route (VLAN, MAC) ECMP count=2 Adj base=198 BGP route points to adjacency table for fast update Support urpf(strict and loose) {VRF, IP} LPM Table (Summary Routes) / / / / /16 (VLAN, MAC) ECMP count=2 Adj base=200 ECMP count=2 Adj base=200 Adjacency Table(64K) 198 VLAN-1 MAC VLAN-2 MAC VLAN-3 MAC VLAN-4 MAC-4 38

39 UPC Forwarding Lookup FC/FCoE Related Tables N6k-3 VE N6k-1 FCF FCF CNA FCoE FCID 0x2e0001 N6k-4 N6k-2 FCID 0x Same lookup tables for FC and FCoE FCID Table Store FCID for local initiator/target 0x stored on N6k-2. 0x2e0001 stored on N6k-4 Share space with MAC table FC Routing Table Store routes to reach other FCF or FC switch 4K entries HW Zoning Table Security policy for SAN. Similar to ACL in IP world 4K-entry zoning table per UPC. Distributed among UPC. One UPC only store zoning entries for hosts attached to this UPC(see next slide) 39

40 FC/FCoE Zoning Table Zoning policing is distributed on UPC Each UPC stores the policing related to the hosts connected to it. Zoning entries per switch not limited by 4k zoning table 0x2e0001-0x x2e0002-0x zoneset name ZS_1 vsan 1 zone name Z_windows vsan 1 * fcid 0x2e0001 * fcid 0x zone name Z_linux vsan 1 * fcid 0x2e0002 * fcid 0x x FCoE UPC 0x2e0001 UPC 1 UPC 2 0x2e0002 0x x2e0001 0x x2e0002 0x FCoE 40

41 QSFP Transceivers and Cables

42 Bit Spray Over QSFP-40G Cisco Nexus 6004 and all its components are designed to carry 40-Gbps flow. The bit spray depicted here applies to QSFP interfaces running at 40 GE mode. When the QSFP interface is configured as four 10 GE interfaces, each fiber and copper cable will carry the packet for one 10 GE interface. Transmit Receive 64b/66b Encoder Stream of 66-bit words Physical Coding Sub-layer (Round robin distribution of 66-bit words) PCS Lane 1 PCS Lane 2 PCS Lane 3 PCS Lane 4 PCS Lane 1 PCS Lane 2 PCS Lane 3 PCS Lane Physical Coding Sub-layer Stream of 66-bit words b/66b Decoder 42

43 QSFP-40G-SR4 40G-SR4 40G-SR4 MPO-12 MPO-12 40G-SR4 10G-SR 10G-SR 10G-SR 10G-SR Support 100M with OM3 MMF and 150M with OM4 MMF MPO-12 LC Breakout 43

44 QSFP-40G-CSR4 40G-CSR4 MPO-12 MPO-12 40G-SR4/CSR4 Support 300M with OM3 MMF and 400M with OM4 MMF Compatible with Cisco QSFP- 40G-SR4 Compatible with 10G SFP-SR 40G-CSR4 10G-SR 10G-SR 10G-SR 10G-SR MPO-12 LC Breakout 44

45 Multimode Ribbon Fiber 40 GE For QSFP-40G-SR4 and QSFP-40G-CSR4 12-Fiber MPO Connector MPO/MTP connector with 12 pins Use 4 fiber pairs 4 TX and 4 RX allows for 40 GE 4 unused fibers in the center 45

46 QSFP-40G-LR4 QSFP-40G-LR4 can only be connected to each other. 10KM with Single mode fiber. Can t connect to 10Gbase-LR 4x ROSA Multiplexer Host Card TIA 4x TOSA LC LC IEEE Standard 40GBASE-LR4 46

47 Physical Connection Choices 40G QSFP+ Interfaces QSFP-LR4 QSFP-LR4 40G QSFP+ Interfaces 40G QSFP+ Interfaces QSFP-SR4/CSR4 QSFP-SR4/CSR4 40G QSFP+ Interfaces 40G QSFP+ Interfaces 1M, 3M, 5M Passive 7M, 10M Active 40G QSFP+ Interfaces 40G QSFP+ Interfaces QSFP-SR4/CSR4 10G SFP+ Interfaces 40G QSFP+ Interfaces QSFP-4x10G 10G SFP+ Interfaces 47

48 FET-40G Support from 6.0(2)N2(1) Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004 Supported on Cisco Nexus 6004 and Nexus 2248PQ-10G Interoperable with FET-10G Support for 100m distance with OM3 Cisco Nexus 6000 Cisco Nexus 6000 FET-40G FET-40G FET-40G Cisco Nexus 2248PQ-10G FET-10G Cisco Nexus 2232PP and 2232TM-E 2232TM Cisco Nexus 2248TP-E 2248TP 48

49 N6K Supported Transceiver QSFP Optical Transceiver Type QSFP-40GE-LR4 QSFP-40G-SR4 QSFP-40G-CSR4 FET-40G-6.0(2)N2(1) Distance 10KM with SMF *ethernet only 100m with OM3 150m with OM4 300m with OM3 400m with OM4 100m with OM3 Complete QSFP support matrix for N6k and other platforms can be found here: r_modules/compatibility/matrix/ol_24900.pdf QSFP Twinax Copper Cable Type Distance QSFP-4x10G-1M QSFP-4x10G-3M QSFP-4x10G-5M QSFP-4x10G-AC7M QSFP-4x10G-AC10M QSFP-H40G-1M QSFP-H40G-3M QSFP-H40G-5M QSFP-H40G-AC7M QSFP-H40G-AC10M 1m 3m 5m 7m 10m 1m 3m 5m 7m 10m 49

50 SFP+/QSFP+ Transceiver Interoperability Matrix 10GBase- SR Cisco QSFP SR4 Cisco QSFP CSR4 IEEE QSFP SR4 Cisco FET-40G Cisco FET- 10G 10GBase-SR *1 Cisco QSFP SR4 Cisco QSFP CSR4 *1 IEEE QSFP SR4 Cisco FET-40G Cisco FET-10G *1 IEEE 40GBase-SR4 Max Launch Power specification (+2.4dBm) exceeds the overload specification for 10GBase-SR Receiver (-1 dbm) and for QSFP-CSR4 Receiver (-1dBm) 50

51 Traffic Distribution with QSFP being 40G Interface Transmit Four Fibers in MTP/MPO cable Receive QSFP-40G-SR4 QSFP-40G-SR4 QSFP as 40GE QSFP-40G-CSR4 QSFP-40G-CSR4 QSFP as 40GE Four wavelengths in one single mode fiber QSFP as 40GE QSFP-40G-LR4 QSFP-40G-LR4 QSFP as 40GE 51

52 Traffic Distribution with QSFP being 4x10G Interfaces Transmit QSFP-40G-SR4 QSFP-40G-CSR4 Receive Four Fibers in MTP breakout cable SFP-10G-SR SFP-10G-SR SFP-10G-SR QSFP as four 10G SFP-10G-SR Four wavelengths in one single mode fiber QSFP-40G-LR4 QSFP-40G-LR4 QSFP as four 10G QSFP as four 10G 52

53 QSA-QSFP to SFP Adapter Support from 6.0(2)N2(1) Convert one QSFP port to one SFP+ port. Provides 1G support on Nexus 6004 Long reach connection between N6004 and 10G device Following 1G/10G transceiver will be qualified with QSA in Harbord+ release-6.0(2)n2(1) 10G-LR 1G GLC-T, 1G SX, LX Requires a mode change on QSFP port to 4x10G Only first 10G interface can be used. Remaining three 10G shown as Link Not Connected QSFP+ Adapter 1G SFP or 10G SFP+ Transceiver 53

54 Interface Speed Mode By default QSFP port is in 40 GE mode. Port speed can be changed at a group of three QSFP ports(n6004) or group of two QSFP(N6001). The group of 12 QSFP ports on N6004 needs to be reset after port mode change. The fixed 48 x QSFP is named in the same way as the ports on the expansion module. Eth8/ GE Interface Ethernet 8/1 10 GE Interface Ethernet 8/1/1 Ethernet 8/1/2 Ethernet 8/1/3 Ethernet 8/1/4 Eth1/

55 Convert 40GE Interface to Four 10 GE Interfaces Nexus 6004 Convert the QSFP Interfaces to Four 10 GE Interfaces When Connecting to a SFP+ Port or to the QSFP Interfaces That Operate in Four 10 GE Mode, such as the QSFP Uplink of N2248PQ-10G or the Cisco Nexus 5500 QSFP GEM. 1 2 Apply global CLI to change interface types to 10 GE. Every three contiguous QSFP interfaces resides on one UPC ASIC. The port range specified in the CLI has to include all ports on the ASIC. Power off the affected modules. Every group of 12 QSFP interfaces is managed as one module, even for the fixed interfaces. N6004(config)# interface breakout slot 1 port 1-6 map 10g-4x N6004(config)# poweroff module? <1-8> Please enter module number N6004(config)# poweroff module 1 3 Power on the affected modules. N6004(config)# no poweroff module 1 N6004(config)# 55

56 Convert 40GE Interface to Four 10 GE Interfaces Nexus 6001 Convert the QSFP Interfaces to Four 10 GE Interfaces When Connecting to a SFP+ Port or to the QSFP Interfaces That Operate in Four 10 GE Mode, such as the QSFP Uplink of N2248PQ-10G or the Cisco Nexus 5500 QSFP GEM. 1 2 Apply global CLI to change interface types to 10 GE. For N6001 two contiguous QSFP interfaces resides on one UPC ASIC. The port range specified in the CLI has to include all ports on the same UPC. Power off the affected modules.* All four QSFP ports of N6001 resides on same module N6001(config)# interface breakout slot 2 port 1-2 map 10g-4x N6001(config)# poweroff module? <1-2> Please enter module number N6001(config)# poweroff module 2 3 Power on the affected modules. N6001(config)# no poweroff module 2 * The requirement of power cycle module after speed change is limited by software 56

57 QSFP Cabling Infrastructure Parts Reference Joint whitepaper with Corning for Nexus 6000 Cabling Infrastructure Joint whitepaper with Panduit for Nexus 6000 Cabling Infrastructure MTP Panel MTP to LC Panel MTP Jumper MTP to 8 LC breakout MTP Trunk( Fiber) 57

58 QSFP to QSFP Connection with Structured Cable Reference MTP Trunk provide fibers for multiple 40G connection 58

59 QSFP to SFP Connection with Structured Cable Reference Use MTP to 8 LC breakout cable or use MTP to LC panel 59

60 Direct Connection(QSFP-QSFP or QSFP-SFP) Reference 60

61 Nexus 6000 Access Control List

62 ACL Types and Features Security ACL ACL for QoS classification and Policing PBR (Starting from 6.0(2)N2(1)) User configured ACL to redirect traffic ACL for SPAN/ERSPAN (roadmap feature) To identify, the traffic needs to be mirrored; not userconfigurable with 6.0(2) ACL for control traffic To redirect control traffic to CPU CoPP 62

63 Security ACLs interface Ethernet1/5 ip port access-group PACL-1 in switchport access vlan 100 vlan access-map VACL-map match ip address VACL-1 action forward vlan filter VACL-map vlan-list 100 interface Vlan100 no shutdown ip access-group RACL-1 in ip access-group RACL-2 out ip address /24 ACL Type Attach Point Direction PACL L2 interface, L2 PortChannel IN VACL VLAN IN OUT RACL L3 interface, L3 PortChannel, sub-interface SVI IN OUT PACL: Port ACL Ingress Direction VACL: VLAN ACL for Both Ingress and Egress; Apply to Preand Postrouted Traffic RACL: Routed ACL; Apply Only to Traffic That Requires Routing; Direction Indicated in the CLI 63

64 4096 Entries ACL Scaling and TCAM Partition 4096 ACE entries per UPC are organized in blocks of 64 ACE entries. 6.0(2)N2(1) release provides CLI to change the size of each region. Each IPv6 ACL (without port range) requires two TCAM entries. Default TCAM Partition VACL(1024) IFACL(1152) QoS(448) RBACL(1024) SPAN(64) Control Traffic(256) N6000-TME1# sh platform afm info tcam 0 region ifacl ifacl tcam TCAM configuration for asic id 0: [ vacl tcam]: range [ifacl tcam]: range * [ qos tcam]: range [rbacl tcam]: range [ span tcam]: range [ sup tcam]: range TCAM [ifacl tcam]: [v:1, size:1152, start:1024 end:2175] In use tcam entries: 24 TCAM Region , Usage 64

65 4096 Entries Security ACL-to-TCAM Region Mapping RACL is placed into different regions to allow parallel lookup to apply multiple ACLs for same packet (for example, PACL, VACL, and RACL for SVI at the Default same TCAM time). Partition VACL(1024) IFACL(1152) QoS(448) Ingress and egress VACL Ingress PACL Ingress RACL for L3 interface Ingress RACL for sub-interface of physical port Egress RACL for all types of L3 interfaces RBACL(1024) SPAN(64) Control Traffic(256) Egress CTS (future) Ingress RACL for SVI Ingress RACL for L3 PortChannel Ingress RACL for PortChannel sub-interface 65

66 Sharing ACE When the same ACL policy (security ACL such as PACL, VACL, RACL, and QoS ACL) is applied to multiple interfaces or VLAN, only one copy is stored in TCAM. The same ACL rules are shared by multiple interfaces and VLANs. Each ACL policy has a label. By assigning the same label to multiple interfaces and VLANs, you can apply the same TCAM rule to multiple interfaces or VLANs. interface Ethernet1/10 ip port access-group ip-list-1 in interface Ethernet1/11 ip port access-group ip-list-1 in interface Ethernet1/12 ip port access-group ip-list-1 in Label xyz xyz xyz xyz eth1/10 eth1/11 eth1/12 IPV4 ACL ip-list-1 10 permit ip / /24 20 permit ip / /24 30 permit tcp / /24 range

67 ACL Programming Scope PACL and RACL for L3 interface and sub-interface of L3 interface are programmed only on the UPC where the port resides. VACL, PACL for PortChannel, RACL for L3 PortChannel, PortChannel sub-interface, and SVI are programmed on all UPC in the chassis. 67

68 ACL Layer 4 Operators There are sixteen L4 operators (L4ops) for port range on UPC. They are shared between UDP and TCP and are shared between ingress and egress and among various type of ACLs. L4ops for port range allows the system to burn one ACE to support ACL with port range (such as gt, lt, range, and neq) There are eight L4 operators for TCP flag on per UPC 12 L4 selectors for UDP and 12 L4 selectors for TCP. Each L4 selector is used to refer to one L4 operator. This implies that each protocol (UDP or TCP) can use up to 12 L4 operators. UDP can use up to 12 L4 operators for port range. TCP can use up to 12 L4 operators for port range and TCP flag combined permit tcp / /24 range Permit udp / /24 range Permit tcp / /24 syn L4 selectors for TCP 12 L4 selectors for UDP syn 8 L4 operators for TCP flag L4 operators for port range 68

69 ACE Expansion Software turns one ACL rule with UDP and TCP port range to multiple ACE entries in TCAM. In such way, the port range is supported without burning L4ops. Each TCAM field has an associated mask including the UDP or TCP port field. The mask can be used to support UDP and TCP port ranges. User-Configured ACL permit tcp / /24 range Rule Software Expands the Rule to Multiple ACEs Action Src IP Dst IP Protocol Src port Dst port Content Permit TCP 0 0 Mask Content Permit TCP 0 64 Mask Content Permit TCP 0 96 Mask Content Permit TCP Mask Port Range 0-63 Port Range Port Range Port Range

70 CLI to Display Expansion Result N6000-TME1# test hardware access-list lou expansion range LOP range expanded to 17 entries index mask port xffff/0x xfff8/0x xff80/0x xff00/0x xfe00/0x xffff/0x xfffe/0x xfffc/0x xfff8/0x xfff0/0x xffe0/0x xffc0/0x xff80/0x xff00/0x xfe00/0x xfc00/0x xf800/0x

71 ACE Expansion Threshold The ACE expansion threshold determines whether software will expand the ACL rule to multiple ACEs or use L4ops to support the port range. The default threshold is 5, meaning that software will turn one ACL line with port range to multiple ACEs if the number of required ACEs after expansion is 5 or fewer. CLI is provided to change the default threshold. Changing the threshold requires chassis reload. Set the expansion threshold to 0 to effectively disable the expansion. N6004(config)# hardware access-list lou resource threshold? <0-32> Enter value for threshold 71

72 ACL Statistics 4096 statistics counters on each UPC Each statistics counter has an associated policer. The policing feature (ingress, egress policing, and storm control) will consume the statistics counter resource. 128 entries are reserved for control traffic; the rest are shared. The statistics counter and policer are allocated on the first-come, first-served basis. Features require statistics counter Ingress and egress policing (future software feature) Unknown unicast, multicast, and broadcast storm control ACL statistics N6000-TME1(config)# ip access-list PACL-1 N6000-TME1(config-acl)# statistics per-entry 72

73 ACL Logging Logs the data packet that hit ACL rules for security reasons Supported for all types of ACLs Logs only the packet that hit the deny rule Vty ACL can log the packet that hit the permit rule 73

74 Nexus 6000 SPAN

75 Cisco Nexus 6000 SPAN Hardware Features SPAN Features Cisco Nexus 6000 Total SPAN sessions Local SPAN sessions 31(16 with current software) 31(16 with current software) ERSPAN sessions 16 Prioritize data over SPAN Line-rate SPAN throughput ERSPAN destination session* ERSPAN with 1588 timestamp* Truncated SPAN and ERSPAN ACL filter for SPAN and ERSPAN* SPAN on drop* SPAN on high latency* SPAN with multiple destination ports* Yes(through scheduling) Yes Yes Yes Yes Yes Yes Yes Yes (Each destination port burns one SPAN session.) *Software roadmap features 75

76 Cisco Nexus 6000 SPAN Differentiators Large number of active SPAN sessions 31 bidirectional active SPAN sessions supported by hardware 16 bidirectional active SPAN sessions supported by software Line-rate SPAN Line-rate SPAN for multiple 40 GE ports Built-in extra capacity in fabric to accommodate SPAN traffic Intelligent SPAN Prioritize data over SPAN in case of fabric link oversubscription due to SPAN traffic Conditional SPAN ERSPAN with 1588 timestamp 76

77 SPAN Features Supported by Software 6.0(2)N1(2) 16 bidirectional active SPAN sessions ERSPAN source session SPAN source port can be: RX or TX or both directions for interface, PortChannel RX for VLAN Up to 128 SPAN source port/vlan per SPAN session One SPAN destination port per session Truncated SPAN/ERSPAN 77

78 Ingress SPAN Packet Flow Data is replicated at ingress port ASIC-Unified Port Controller (UPC). SPAN packets are queued at the SPAN destination port VOQ. Ingress Interface (RX SPAN source) Egress Interface Data Span Unicast VOQ Multicast VOQ 224Gbps Data Span Unified Fabric Controller Data SPAN Destination Span 78

79 Egress SPAN Packet Flow SPAN copy is made at egress pipe of the TX SPAN source port. SPAN packets are looped back to ingress pipe of UPC and sent to switch fabric. Ingress Interface Unicast VOQ Data Egress Interface (TX SPAN source) Data Unicast VOQ Span Data Multicast VOQ 224Gbps Unified Crossbar Fabric Span SPAN Destination 79

80 High-Performance SPAN No penalty to production data traffic with SPAN Extra bandwidth capacity built in for SPAN traffic Line-rate SPAN throughput with extra fabric link bandwidth First product in the market to support 16 line-rate SPAN sessions Scheduler Prioritizes Production Data Extra Fabric Bandwidth for SPAN 448 Gbps 120-Gbps Data 120-Gbps Data Data Span 224 Gbps Unified Crossbar Fabric 448 Gbps Separate Buffer Pool for SPAN 80

81 Conditional SPAN SPAN on Drop Copy the dropped packet to capture device. Data Flow 1/8 Nexus /10 Correlate the packet drop with application Work for unicast packets only Future software roadmap feature N6004# sh monitor session 1 session type : span-on-drop state : up acl-name : acl-name not specified source ports : Eth1/8 Eth1/9 destination ports : Eth1/11 Data Flow 1/9 1/11 SPAN-on-drop packets Capturing Device 81

82 Line-Rate Multicast with Line-Rate SPAN 240-Gbps replication capacity for each UPC 120-Gbps for multicast; 120-Gbps for SPAN 240-Gbps Replication per UPC Multicast SPAN I X I A 40GE Multicast Source Ingress UPC 224 Gbps Unified Crossbar Fabric 448Gbps 448Gbps Egress UPC Egress UPC Receiver 40GE Receiver 40GE Receiver 40GE SPAN Destination SPAN SPAN I X I A I X I A 82

83 Nexus 6000 Multicast

84 Cisco Nexus 6000 Multicast Features Summary As of Release 6.0(2)N1(1) IGMP snooping for IGMPv1, v2, and v3 PIM-SM PIM-SSM without vpc and PIM-SSM with vpc+ PIM-BiDir without vpc and PIM-BiDir with vpc+ PIM Policy MSDP Anycast RP with PIM (RFC 4610) Anycast RP with MSDP 8000 IGMP snooping entries 8000 mroutes with vpc, mroute without vpc supported by software 84

85 Cisco Nexus 6000 Multicast Highlights High Performance Line-rate L2 and L3 multicast throughput with all frame sizes Low latency at scale High Scalability 64K HW entries for mroutes and IGMP snooping tables; 8,000 mroutes and 8,000 IGMP snooping groups supported at FCS Same architecture from 64 to 384 ports Larger buffer for burst absorption Optimized multicast replication Fabric replication and egress replication Enhanced features IP-based forwarding for IGMP snooping BiDir PIM support Flow-based hashing for multicast over PortChannel Better traffic visibility 85

86 Efficient Multicast Replication Optimized multicast replication Fabric replication and egress replication; one copy is replicated to egress UPC, where there is a receiver Line-rate multicast replication Egress UPC VLAN 10 VLAN 10 Ingress UPC Unified Crossbar Fabric Egress UPC V 10 V 20 VLAN 10 VLAN 10 VLAN 20 86

87 Multicast VOQ Unicast VOQ Multicast VOQ 8000 multicast VOQs to eliminate HOLB and help ensure high throughput Tracks the fan-out of the egress UPC; packets with different egress UPC fanouts are assigned to different VoQs so that it can be scheduled and served independently Ingress UPC Egress UPC Unified Crossbar Fabric Egress UPC Egress UPC

88 Multicast VOQ Unicast VOQ Multicast Buffering and Congestion Control Multicast packets are buffered and dropped at egress. Multicast packets are dropped for the port when the multicast queue depth reaches threshold, avoiding buffer starvation for the rest of the ports and maintaining fairness. Maintain lossless fabric for unicast. Ingress UPC 16MB Unified Crossbar Fabric Egress UPC Multicast Flow control Unicast Queue Unicast Port 1 Multicast Queue Unicast Queue Port 2 Multicast Queue 88

89 Multicast VOQ Unicast VOQ Multicast Hashing over PortChannel N6004 implements flow based hashing for multi-destination traffic and it support multicast load sharing over PortChannel with 5-tuple packet header. Traffic is replicated to all egress UPC where PortChannel member resides Egress UPC run hash calculation and one egress port is chosen to send out multicast packets. The UPC ASIC that is not supposed to send out packet will drop packet (Egress UPC 1 in the example) Egress UPC 1 Ingress UPC Port 3 Selected Port 1 Port 2 Unified Crossbar Fabric Egress UPC 2 Port 3 Selected Port 3 Port 4 89

90 RPF Check Failure Packets that fail RPF check are copied to CPU to generate PIM assert. When IIF is L3, redirect to CPU. When IIF is SVI, copy to CPU while forwarded in the incoming VLAN. A bloom filter is implemented so that no more than one packet from small flow is copied to CPU. The mechanism can expedite the process of assertion and reduce CPU workload. A flow is hashed to an entry in the bloom filter table. A flag is set for this entry when one packet is copied to CPU. The subsequent packets will not be copied to CPU when this flag is set. Software clears this flag periodically to allow more packets from this flow to be copied to CPU. 90

91 CPU-Bound Multicast Data Packet Processing Multicast data packets are copied to CPU to create multicast forwarding state. First-hop router: Source registration Last-hop router: Share tree to source tree switchover RPF check failure packets: For PIM assert These three types of packets will be rate-limited by three different policers. It doesn t have the Trident s problem, where all these packets are rate-limited by the same policer. 91

92 IP-Based Forwarding for IGMP Snooping Source IP and group address-based forwarding for IGMPv3 snooping even when Cisco Nexus 6000 is L2 switch Can filter traffic based on source IP No concern of overlapping multicast MAC addresses H1 Multicast MAC based forwarding Vlan E eth1/1 v3 Report ( ) Group: Include: IP-Based Forwarding Vlan eth1/1

93 Increase Software Table Size for Multicast Change the SOFTWARE table size limitation. No change to hardware table partition Software stops learning additional mroute or IGMP snooping entries once the configured limit is reached. Software supports 8000 mroute with vpc and mroute without vpc Release 6.0(2)N2(1) will increase IGMP snooping group limit to N6004(config)# hardware profile multicast max-limit? < > Mcast Table Entries N6004(config)# hardware multicast snooping group-limit? < > Specify a value between Increases mroute table size in software. Default is 8000 Increases IGMP snooping table size in software. Default is

94 FabricPath and vpc+ Benefits Eliminate STP between access and aggregation vpc + allow flexibility with topology and scale L3 Core Unicast and multicast peering allowed on vpc+ No need to configure the orphan port suspend Support PIM-SSM and PIM-BiDir Better mroute scalability and simplified configuration L3 L2 VPC+ Domain VPC+ Domain FabricPath core port 16k mroute with vpc+. 8k mroute with vpc No need to configure special VLAN with bind-vrf 94

95 Nexus 6000 Quality of Service

96 Cisco Nexus 6000 QoS Features Eight classes of service; 2 reserved for control traffic, 6 for data traffic Traffic classification DSCP, CoS, and ACL Strict Priority Queuing and DWRR DCBX 802.1Qaz Packet marking DSCP, CoS, and ECN Ingress and egress policing 4096 policers per ASIC No drop system class Flexible buffer management 96

97 Cisco Nexus 6000 QoS Processing Flow Ingress UPC Trust CoS/DSCP L2/L3/L4 info with ACL If Buffer Usage Crosses Threshold: Tail drop for drop class Assert pause signal to MAC for no-drop system class VoQs for Unicast (8 per egress port) MAC MAC Traffic Classification Ingress Cos/DSCP Marking Ingress Policing PAUSE ON/OFF Signal ECN Marking MTU Checking Truncate or Drop Packets if MTU is Violated Egress Policing Per-class Buffer Usage Monitoring Egress Scheduling Multicast Queues Proxy Queues Egress Queues Unicast Crossbar Fabric Egress UPC Strict Priority + DWRR Scheduling Multicast 97

98 Packet Classification Packet can be classified based on the DSCP, IP Precedence, 802.1p, or ACL. Default TCAM carving allocates 448 ACE entries for QoS classification. Packet classification takes place at ingress. Packet marking is trusted by default. 98

99 Multicast VOQ Unicast VOQ Increased Packet Buffer 25-MB packet buffer is shared by every three 40 GE ports or twelve 10 GE ports. Buffer is 16 MB at ingress and 9 MB at egress. Unicast packet can be buffered at both ingress and egress. Multicast is buffered at egress. Ingress UPC Egress UPC 16MB 224 Gbps Unified Crossbar Fabric 448 Gbps 9MB 99

100 Flexible Buffer Management Ingress Buffer Shared buffer is good for burst absorption. Dedicated buffer is good for predicated performance for each port. Buffer management is flexible: Dedicated plus shared. Long-distance FCoE, video editing (i.e., AVID), Big Data, and distributed storage UPC Ingress Buffer (16MB) Port 1 Dedicated SPAN Port 2 Dedicated Control Shared Packet Buffer Port 3 Dedicated 100

101 Tune Buffer Allocation at Ingress queue-limit under network-qos policy specifies the dedicated buffer for each port and each class. Total ingress buffer minus the dedicated buffer and buffer for control and SPAN will be in the shared buffer pool. N6004-TME2(config)# policy-map type network-qos Policy-buffer N6004-TME2(config-pmap-nq)# class type network-qos class-default N6004-TME2(config-pmap-nq-c)# queue-limit bytes N6004-TME2(config-pmap-nq-c)# system qos N6004-TME2(config-sys-qos)# service-policy type network-qos Policy-buffer 101

102 Flexible Buffer Management Egress Buffer 9-MB packet buffer at egress UPC is shared among three 40 GE or twelve 10 GE. CLI is provided to allocate buffer between unicast and multicast (future). Unicast traffic can be buffered at egress and ingress. Multicast is buffered at egress in case of interface oversubscription. UPC Egress Buffer (9MB) Unicast Buffer Multicast Buffer 102

103 Unicast Flow Control and Buffering Unicast is queued at egress buffer pool first and then ingress buffer after egress queue is full. Take advantage of ingress buffer from multiple port or ASIC for unicast burst absorption. Ensure fairness among multiple ingress ports 103

104 Multicast VOQ Unicast VOQ Multicast Buffering Management Multicast packets are buffered at egress. Egress multicast queue drop threshold determines the amount buffer for multicast traffic Future software will provide CLI to tune this multicast queue drop threshold Ingress UPC 16MB Unified Crossbar Fabric Egress UPC Multicast Flow control Unicast Queue Unicast Port 1 Multicast Queue Unicast Queue Port 2 Multicast Queue 104

105 Long Distance FCoE The longer the distance the more time it takes for PAUSE frame to arrive at upstream switch Buffer required for FCoE is proportional to speed and distance Roughly 1MB is required for 10G FCoE for 10KM Future software to support FCoE for more than 300m PAUSE frame arrived at T2 Send PAUSE frame at T1 FCoE PAUSE Threshold Receiving queue Sender FCoE data stream Receiver 105

106 Identify Packet Drop Due to Congestion switch# show interface ethernet 1/7 Ethernet1/1 is down (SFP not inserted) Dedicated Interface <snip> RX <snip> 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble input discard 0 Rx pause TX unicast drop at ingress 0 unicast packets 0 multicast packets 0 broadcast packets 0 output packets 0 bytes 0 jumbo packets 0 output error 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause Broadcast/Multicast/unknown unicast flooding drop at egress switch# show queuing interface ethernet 1/7 Ethernet1/1 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 100 RX Queuing qos-group 0 Dedicated buffer for this class q-size: , HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: 0 Statistics: Pkts received over the port : Ucast pkts sent to the cross-bar : Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 0 Pkts discarded on ingress : Per-priority-pause status : Rx (Inactive), Tx (Inactive) ingress drop for each class 106

107 Identify Congested Egress Port with VOQ Counters I X I A I X I A 1/7 1/10 Ingress UPC 3 Ingress UPC 4 Unified Crossbar Fabric Egress UPC 3 N6004# show platform software qd info counters voq interface ethernet 1/ slot asic TRANSMIT TAIL DROP HEAD DROP QUEUE QUEUE QUEUE /8 I X I A Egress port 107

108 Identify Congested Egress Port with VOQ Counters N6004# show platform software qd info counters voq asic-num port TRANSMIT TAIL DROP HEAD DROP Eth1/8 QUEUE Congested egress port and class of service N6004# show platform software qd info counters voq asic-num port TRANSMIT TAIL DROP HEAD DROP Eth1/8 QUEUE Ingress UPC number 108

109 Map Interface to UPC Number N6004# show hardware internal bigsur all-ports Bigsur Port Info: Port asic inst inst name idx slot asic eport logi flag adm opr if_index diag ucver sup b3 en dn pass 0.00 sup b3 en dn pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 dis dn 1a pass gb1/ b3 dis dn 1a pass gb1/ p 6 b3 en up 1a pass gb1/ p 7 b3 en up 1a pass gb1/ p 8 b3 en up 1a pass gb1/ p 9 b3 en up 1a pass gb1/ b3 dis dn 1a07a000 pass gb1/ p 11 b3 en up 1a07b000 pass 0.00 Done. 109

110 Nexus 6000 Switch Fabric Mode

111 What Is Switch Fabric Mode? Switch fabric mode determines the speed of a single fabric link between UPC and Fabric. When fabric mode is 10 GB, a single fabric link between UPC and Fabric runs at 14 Gbps. With fabric mode of 40 GE, a single fabric link between UPC and Fabric runs at 56 Gbps. All Fabric fabric links have to run at the same mode or the same speed. In other words, the fabric mode is a system wide configuration. Physical wire connections remain the same for both fabric modes. The fabric mode doesn t change the total bandwidth between UPC and Fabric. 111

112 Cisco Nexus 6000 Switch Fabric Mode Configuration, done through the CLI, is system wide. No hardware change is necessary. Switch fabric mode determines the speed of a single fabric link between UPC and Fabric. Switch fabric mode is independent of front panel port speed Default switch fabric mode : 40G for Nexus 6004 and 10G for Nexus GE Switch Fabric Mode 10 GE Switch Fabric Mode 8x56Gbps 32x14Gbps UPC 4x56Gbps UPC Fabric Links Fabric Links 16x14Gbps 112

113 Nexus 6000 Switch Fabric Mode Recommendation Switch fabric mode provides flexibility Use 40G switch fabric mode on N6004 and N6001 when there are mixed 10G and 40G ports and throughput between 40G ports is more important than latency Use 10G switch fabric mode to achieve lowest possible latency among 10G ports N6004(config)# fabric-mode 40g Fabric mode set to 40G. Please copy the configuration and reload the switch N6004(config)# 113

114 Switch Fabric Mode and Latency With 40 GE fabric, traffic from 10 GE to 10 GE may fall back to store-andforward with a heavy traffic load. We recommend 10 GE fabric when most of the ports are 10 GE and latency is important. 40 GE Fabric Mode Ingress 10 GE Egress 10 GE 40 GE Cut-through and store-and-forward 40 GE Cut-through Cut-through 10 GE Fabric Mode Egress Ingress 10 GE Cut-through 40 GE Cut-through 10 GE 40 GE Store-andforward Store-andforward Store-andforward 114

115 Switch Fabric Mode and Throughput 40 GE switch fabric mode Support for 40 GE flow Line rate with different port speed combination 10 GE switch fabric mode Cannot support >10-Gbps flow Line rate between 10 GE and 40 GE interfaces (no dependency on load balance) Throughput between 40 GE ports depends on hashing result 10G Fabric Mode 40G Fabric Mode 10 GE(Egress) 40 GE(Egress) 10 GE(Egress) 40 GE(Egress) 10 GE (ingress) Line Rate 40 GE line Rate Line rate 40 GE Line rate 40 GE (ingress) Line Rate <14Gbps flow Throughput depends on hashing results. Line rate Line rate 115

116 Nexus 6000 Applications Compact Aggregation Large Scale Fabric (Layer 2 or Layer 3) Nexus 7000 CORE Nexus7000 Nexus 6004 Neuxs 6004/ Nexus 6001 Nexus 2000 L3 VPC to Nexus 7000 or 6004 AGG. ACCESS FEX Nexus 7000 Nexus 6004 Nexus 6004/ Nexus 6001 L3 FabricPath CORE SPINE LEAF High Performance Computing (HFT) Multi hop FCoE Nexus 6004 CORE/AGG. Nexus 7000 CORE FCoE A FC B L3/L2 Fabric Nexus 7000 Nexus 6004 AGG Nexus 3548 Nexus 6001/6004 ACCESS Nexus 6001 Access FCoE FC BRKDCT

117 Nexus 6000 Key Takeaways Architecture Rich Feature set Performance Scalability Innovations Key building block for Dynamic Fabric Automation STP/vPC/FabricPath, FEX, L3, VM-FEX/Adapter- FEX Line rate L2 and L3 with low latency 256K MAC/128K ARP/32K LPM routes, 16K bridge domain 4K VRF Intelligent SPAN, Buffer monitoring, Latency monitoring 117

118 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Don t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit 118