Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture

Transcription

1

2 Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture Nicolas Delecroix Technical Marketing Engineer Data Center Group Nexus Business Unit

3 Nexus 5000 Evolution Nexus 5020 Customers 56-Port, 2RU Nexus Port, 1RU Nexus 5596T 10Gbase-T, 2RU Nexus Port, 1RU Nexus Port, 2RU 3

4 Introducing Cisco Nexus 5600/6000 Nexus 6004 Nexus 5020 Customers 56-Port, 2RU Nexus Port, 1RU Nexus 5596T 10Gbase-T, 2RU 96 port 40G, 4RU Nexus Port, 1RU Nexus Port, 2RU 4

5 Introducing Cisco Nexus 5600/6000 Nexus port 10G, 6 port 40G, 1RU Nexus Nexus port 10G, 8 port 40G, 2RU Nexus 5020 Customers 56-Port, 2RU Nexus Port, 1RU Nexus 5596T 10Gbase-T, 2RU 96 port 40G, 4RU Nexus Port, 1RU Nexus Port, 2RU 5

6 Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture Nicolas Delecroix Technical Marketing Engineer Data Center Group Nexus Business Unit

7 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 7

8 Nexus 5672 Chassis Port-Side View All ports provide Ethernet and FCoE 16 Unified Ports (orange) also provide 2/4/8G FC L2 and L3 VXLAN 48 fixed 1/10G SFP+ interfaces 6 fixed 40G QSFP interfaces 8

9 Cisco Nexus 5672 Chassis Power Supply and Fans Power Supply 1+1 Redundancy Fan Module 2+1 Redundancy Console Mgmt0 USB 9

10 Nexus Chassis Port-Side View 48 Fixed 10G SFP+ Ports, 4 fixed 40G QSFP+ ports 2 Expansion Slots Expansion module provides 24 Unified ports 10G Ethernet/FCoE or 2/4/8G FC, and 2 ports 40G Ethernet/FCoE 10

11 Cisco Nexus Chassis Power Supply and Fans Power Supply 2+2 Redundancy Console Mgmt0 USB Fan Module 3+1 Redundancy 11

12 Cisco Nexus 6004 Chassis Port-Side View 12 QSFP+ ports Expansion Module N6K-6004-M12Q 4RU 12

13 Cisco Nexus 6004 Chassis Power Supply and Fans Six power supply slots; a minimum of three is required. Supports both and redundancy. Each power supply is rated 1100W, VAC. The chassis has four fan trays. A minimum of three is required. Power Supply Grid Redundancy or Redundancy Console Mgmt0 USB Fan Module 3+1 Redundancy 13

14 Unified Port Line-Card Expansion Module - N6004X-M20UP For Nexus * 10G SFP+ unified ports expansion module 2/4/8G FC port 1G/10G and FCoE Max number of native FC ports per chassis is 48 (as of NXOS 7.0(2)N1(1) release) SFP+ port allows support for a larger variety of optical transceivers Same transceivers supported as on G ports including 10G long distance, 1G copper and optical 14

15 Cisco Nexus 5600/6000 Airflow The units have a port side and a fan side. No front or back The units support both port side exhaust and port side intake Port side exhaust (used for in server rack airflow alignment) Port side intake (used for network rack airflow alignment) Different power supply and fan modules are required for different airflow directions 15

16 N5600/6000 and N5500 Comparison Feature Nexus 5600/6000 Nexus 5500 L2 Throughput Line rate Line rate L3 Throughput Line rate 160Gbps Latency ~1us 1.8us Native 40GE Support Yes 4xQSFP module 10G flows 1GE Support Yes Yes Unified Ports Yes Yes FEX Scaling 48 (6004 L2), 24 (6004 L3, 5600) 24 with L2, 16 with L3 MAC Table 256K (shared with ARP/ND) 32K IPv4 Unicast Routes 32K 16K IPv4 Multicast Routes 32K 8K (shared with ARP) IPv4 ARP/Host Route Entries 128K 16K (shared with IPv6 ND and mroute) 16

17 N5600/6000 and N5500 Comparison (cont.) Feature Nexus 5600/6000 Nexus 5500 IPv6 Unicast Routes 8K (shared with IPv4) 8K (shared with IPv4) IPv6 ND Entry/Host Route Entries 85K (shared with MAC and ARP) 8K (shared with mroute and ARP) IPv6 Multicast Routes 32K 8K (shared with ARP, ND. Limit is with /64 routes) VLAN 4K 4K VRF 4K 1K Segment ID Yes No ACL 4K 4K L4ops for ACL 24 total (16 for UDP/TCP, 8 for TCP flag) 8 for UDP and 8 for TCP Packet Buffer 17 25MB per 3xQSFP (or 12xSFP) ports 2.1MB per 10G port vpc, FabricPath Yes Yes Dynamic Fabric Automation Yes No VXLAN Yes* No *Software roadmap feature 640KB per 10G port

18 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 18

19 Nexus M/1G Fabric Extender Portfolio Nexus 2224TP Nexus 2248TP Nexus 2248TP-E Uplink Ports 2 x 10G SFP+ 4 x 10G SFP+ 4 x 10G SFP+ Host Facing Ports 24 x 100/1000BASE-T RJ45 48 x 100/1000BASE-T RJ45 48 x 100/1000BASE-T RJ45 Oversubscription 1 : : : 1.2 FCoE N/A N/A N/A Buffer 3.8MB 7.6MB 32MB Shared Buffer No No Yes 19

20 Nexus G/10G Fabric Extender Portfolio Nexus 2232PP Nexus 2232TM Nexus 2232TM-E Nexus 2248PQ Uplink Ports 8 x 10G SFP+ 8 x 10G SFP+ 8 x 10G SFP+ 4 x 40G QSFP+ Host Facing Ports 32 x SFP/SFP+ (1/10G) 32 x 1/10GBASE-T RJ45 32 x 1/10GBASE-T RJ45 48 x SFP/SFP+ (1/10G) Oversubscription 1 : 4 1 : 4 1 : 4 1 : 3 FCoE Yes No Yes (30m / Cat6a/7) Yes Buffer 10.2MB 10.2MB 10.2MB 32MB Shared Buffer No No No Yes 20

21 48xFEX Nexus 2000 Fabric Extender Scalability Nexus 5600 and 6000 support all types of FEX, including HP, Fujitsu, Dell, IBM FEX for blade servers. Adapter-FEX and VM-FEX are also supported. Cisco Nexus 5600/6000 Cisco Nexus 2000 FEX + Nexus 5000 Nexus 5500 Nexus 5600 Nexus 6004 Nexus 7000 L L3 x Nexus

22 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 22

23 Nexus x 10G Unicast Throughput Throughput (% of line rate) RFC 2544 Unicast L2 Port Pair RFC 2544 Unicast L3 Port Pair Packet Size (Bytes) 23

24 Nexus x 10G Multicast Throughput Throughput (% of line rate) RFC 3918 Mcast L2 One to Many Accumulated RFC 3918 Mcast L3 One to Many Accumulated RFC 3918 Mcast L2 One to Many Distributed RFC 3918 Mcast L3 One to Many Distributed Packet Size (Bytes) 24

25 Nexus x 10G Unicast 100% Load Latency Average Latency (Nano Seconds) RFC 2544 unicast L2 port pair RFC 2544 unicast L3 port pair Packet Size (Bytes) 25

26 Nexus 6004 Performance Fully Loaded Switch 26

27 Nexus 6004 Performance Spirent Traffic Generator 27

28 Nexus x 40G Unicast Throughput Throughput (% of line rate) RFC 2544 Unicast L2 Port Pair RFC 2544 Unicast L3 Port Pair Packet Size (Bytes) Spirent third party performance report for Nexus

29 Nexus x 40G Multicast Throughput Throughput (% of line rate) RFC 3918 Mcast L2 One to Many Accumulated RFC 3918 Mcast L3 One to Many Accumulated RFC 3918 Mcast L2 One to Many Distributed RFC 3918 Mcast L3 One to Many Distributed Packet Size (Bytes) Spirent third party performance report for Nexus

30 Nexus x 10G Unicast 100% Load Latency Average Latency 1 (microseconds) RFC 2544 Unicast L2 Port Pair RFC 2544 Unicast L3 Port Pair Packet Size (Bytes) 30

31 Nexus x 10G Multicast 100% Load Latency Average Latency (microseconds) RFC 3918 Mcast L2 One-to-Many Accumulated RFC 3918 Mcast L3 One-to-Many Accumulated RFC 3918 Mcast L2 One-to-Many Distributed RFC 3918 Mcast L2 One-to-Many Distributed

32 Nexus x 10G Unicast and Multicast 100% Load Jitter Average Jitter (nanoseconds) Unicast L2 Jitter (ns) Unicast L3 Jitter (ns) Multicast L2 Jitter (ns) Multicast L3 Jitter (ns) Packet Size (Bytes) 32

33 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 33

34 Cisco Nexus 5672 Internal Architecture Switch Fabric Unified Fabric Controller (UFC) UPC 1 UPC 2 UPC 3 UPC 4 UPC 5 UPC 6 Supervisor UPC 0 FC PHY 48xSFP+ 6xQSFP+ CPU 34

35 Cisco Nexus Internal Architecture 24xSFP+ Slot 1 Slot 2 2x QSFP+ 24xSFP+ 2x QSFP+ FC PHY FC PHY FC PHY FC PHY UPC 7 UPC 8 UPC 9 UPC 10 UPC 11 UPC 12 Supervisor Unified Fabric Controller (UFC) UPC 1 UPC 2 UPC 3 UPC 4 UPC 5 UPC 6 CPU 48xSFP+ 2x QSFP+ 2x QSFP+ 35

36 Cisco Nexus 6004 Internal Architecture Switch Fabric UFC 1 UFC 2 UFC 3 UFC 4 UPC 1 UPC 2 UPC 3 UPC 4 UPC 1 UPC 2 UPC 0 FC PHY 12xQSFP+ 20xSFP+ CPU Supervisor 12x40G LEM (M12Q) 20x10G LEM (M20UP) 36

37 Unified Port Controller - UPC ASIC Multimode MAC; built-in PHY for 1, 10, and 40GE Packet parsing and rewriting Lookup engine and access control: L2, L3, FabricPath, ACL, FCoE, and policing Buffering and queuing: Buffer management, PFC for lossless traffic, queuing (Strict Priority Queuing and DWRR), and packet replication (SPAN and multicast) Extra fabric bandwidth for SPAN and multi-destination traffic Front Panel Ports Fabric Connections Twelve 10 GE UPC 224Gbps 448Gbps Three 40 GE UPC 224Gbps 448Gbps 37

38 Cisco Nexus 5600/6000 Packet Processing Flow Ingress Pipeline 1, 10, and 40 GE MAC Parser Security ACL QoS ACL SPAN Filter Policing/Stats Storm Control VLAN Table MAC Table ARP/ND Table IPv4/IPv6 Routing Table FabricPath FC routing Table Forwarding Lookup Ingress ACL Ingress Policing Ingress SPAN Replication Packet Rewrite Ingress UPC Buffer Allocation Buffer Accounting PAUSE Signal Buffer Management Buffer Unicast VOQ (8 per egress port) 8000 Multicast VOQs Unified Crossbar Fabric 38

39 Unified Crossbar Fabric Nexus 5600/6000 Packet Processing Flow Egress Pipeline Egress UPC Unicast buffer Multicast congestion management Unicast Queue Buffer Management Scheduling Multicast Replication Egress ACL Egress Policing* Editing MAC Buffer Multicast Queue TX SPAN Replication *Egress policing is not supported with current software To Ingress Buffer Management 39

40 256,000 Entries Cisco Nexus 5600/6000 Key Forwarding Tables Host table: 256,000 -entry hashing table; actual capacity is slightly less than 256,000 Host table: Shared between MAC, ARP, and ND and /32 host route Host table default carving: 128,000 MAC, 128,000 IP hosts LPM table: 32,000 entries. Also known as summary routes Host Table* LPM Table(32,000) Mroute Table(64,000)* MAC Region Summary Routes (S,G) IP Host Region (ARP/ND/Host route/(*,g)) * Hardware table size. Please check configuration limit for software scaling 40

41 256,000 Entries Cisco Nexus 5600/6000 Host Table Carving In Layer 2, virtualized environments, it can be interesting to have more than 128k MAC addresses on the switch MAC/ARP Hardware Resource Carving allows to change the size of MAC and IP regions to allocate more space for the MAC region. Host Table MAC Region IP Host Region (ARP & ND & /32 Host route) 41

42 Cisco Nexus 5600/6000 Carving Profiles Use of pre-defined carving profiles. HRT = Host Route Table = IP host region. STM = Station Table Management = MAC region. Template Profile hrt-128-stm-128 hrt-96-stm-160 hrt-64-stm-192 hrt-32-stm-224 Description HRT size: 128k, STM size: 128k (default profile) HRT size: 96k, STM size: 160k HRT size: 64k, STM size: 192k HRT size: 32k, STM size: 224k 42

43 Cisco Nexus 5600/6000 MAC/ARP Resource Carving CLI Specify the resource template to use: switch(config)# hardware profile route resource servicetemplate template-name Save the config and reload the switch Show commands: show hardware profile route resource template show hardware profile route resource template default show running-config hardware profile route resource template show startup-config hardware profile route resource template 43

44 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 44

45 QSFP 40G Bidirectional Transceiver 40G deployment with 10G cabling QSFP pluggable 40Gbit/s Bidirectional (BiDi) Transceiver that works with existing Multi-mode fiber infrastructure Using the existing 10Gbit/s Multi-mode infrastructure QSFP-40G-SR-BD has dual LC Connector 40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and received simultaneously on two wavelengths 45

46 QSFP-40G-SR-BD 40G-BD Multimode LC Duplex 40G-BD Supports 100m with OM3 with MMF and 150m with OM4 (OM4+) MMF LC duplex connectors Wavelength: ~850nm and ~900nm Pull-Tab Color: grey 40G-BD Optical DeMux 10G-SR 10G-SR 10G-SR 10G-SR Multimode LC Duplex 46

47 QSFP-40G-SR4 40G-SR4 40G-SR4 MPO-12 MPO-12 40G-SR4 MPO-12 LC Breakout 10G-SR 10G-SR 10G-SR 10G-SR Support 100M with OM3 MMF and 150M with OM4 MMF 47

48 QSFP-40G-CSR4 Supports 300M with OM3 MMF and 400M with OM4 MMF 40G-CSR4 40G-SR4/CSR4 Compatible with Cisco QSFP- 40G-SR4 MPO-12 MPO-12 Compatible with 10G SFP-SR 40G-CSR4 10G-SR 10G-SR 10G-SR 10G-SR MPO-12 LC Breakout 48

49 Multimode Ribbon Fiber 40GE For QSFP-40G-SR4 and QSFP-40G-CSR4 12-Fiber MPO Connector MPO/MTP connector with 12 pins Use 4 fiber pairs 4 TX and 4 RX allows for 40GE 4 unused fibers in the center 49

50 QSFP-40GE-LR4 QSFP-40GE-LR4 can only be connected to each other. 10KM with Single mode fiber. 4x ROSA Multiplexer Host Card TIA 4x TOSA LC LC IEEE Standard 40GBASE-LR4 Can t connect to 10Gbase-LR 50

51 FET-40G Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004 Supported on Cisco Nexus 5600/6004 and Nexus 2248PQ-10G Interoperable with FET-10G Support for 100m distance with OM3 Cisco Nexus 5600/6000 Cisco Nexus 5600/6000 FET-40G FET-40G FET-40G Cisco Nexus 2248PQ-10G FET-10G Cisco Nexus 2232PP and 2232TM-E 2232TM Cisco Nexus 2248TP-E 2248TP 51

52 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 52

53 ACL Types and Features Security ACL MAC, IPv4, and IPv6 ACLs PACL: ACL enabled under L2 interface VACL: ACL enabled for L2 VLAN traffic RACL: ACL enabled for routed traffic RBACL: Role-based ACL for CTS (roadmap feature) ACL for QoS classification and Policing Policy Based Routing (PBR) User-configured ACL to redirect traffic ACL for SPAN/ERSPAN ACL for control traffic To redirect control traffic to CPU, not user-configurable CoPP 53

54 4096 Entries ACL Scaling and TCAM Partition 4096 Access Control Entries per UPC are organized in blocks of 64 ACE entries. Each IPv6 ACL (without port range) requires two TCAM entries. Default TCAM Partition VACL(1024) IFACL(1152) QoS(448) RBACL(1024) SPAN(64) Control Traffic(256) switch# sh platform afm info tcam 0 region ifacl ifacl tcam TCAM configuration for asic id 0: [ vacl tcam]: range [ifacl tcam]: range * [ qos tcam]: range [rbacl tcam]: range [ span tcam]: range [ sup tcam]: range TCAM [ifacl tcam]: [v:1, size:1152, start:1024 end:2175] In use tcam entries: 24 TCAM Region , Usage 54

55 TCAM Carving TCAM Carving allows to change the size of each region Create a template: switch(config)# hardware profile tcam resource template t1 switch(config-tcam-templ)# Set region sizes: switch(config-tcam-templ)# ifacl? < > Enter size of ifacl region (in 64 entry increments) 55

56 TCAM Carving Similarly, other region sizes can be changed: switch(config-tcam-templ)#? ifacl Configure size of ifacl region no Change size to default value qos Configure size of qos region rbacl Configure size of rbacl region span Configure size of span region vacl Configure size of vacl region Display the configuration for a template along with the current usage: switch(config)# show hardware profile tcam resource template name <my_template> 56

57 TCAM Carving switch(config-tcam-templ)# show hardware profile tcam resource template name t Template name: t1 Current state: Created Region Size-allocated Current-size Current-usage Available/free Vacl Ifacl Rbacl Qos Span Sup

58 TCAM Carving Commit a template: switch(config)# hardware profile tcam resource service-template t1 Details of the t1 template you are trying to commit are as follows: Template name: t1 Current state: Committed (Startup/Running) Region Features Size-allocated Current-size Current-usage Available/free Vacl Vacl [ ] Template committed. Please do the following for the template to be applied: 1> Save running config : "copy running-config startup-config" 2> Reboot the switch : "reload 58

59 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 59

60 Why Analytics? A Data Center Today: 1G -> 10G -> 40G -> 100G Virtualized More and more applications Those applications are more and more complex (Hadoop ) More and more flows Latency is very important Ports speed mismatch Buffer discrepancy between the devices With spine/leaf, design is less complex, but there s more speed and more applications 60

61 Switched Port Analyzer (SPAN) Switch Host-A Host-B Monitor Source Monitor Destination Sniffer Device 61

62 Encapsulated Remote SPAN (ERSPAN) Host-B GRE Encapsulated monitored traffic Host-A IP Cloud Monitor Source Monitor Destination Sniffer Device 62

63 Cisco Nexus 5600/6000 SPAN SPAN Features Nexus 5600/6000 Nexus 5500 Total SPAN sessions 31 bi-directional (16 with current NXOS) 4 bi-directional Local SPAN sessions 31 bi-directional (16 with current NXOS) 4 bi-directional ERSPAN sessions 16 4 Prioritize data over SPAN Yes (through scheduling) Yes (SPAN policing) Line-rate SPAN throughput Yes No ERSPAN destination session Yes No ERSPAN with 1588 PTP timestamp Yes* No MTU Truncated SPAN and ERSPAN Yes Yes ACL filter for SPAN and ERSPAN Yes Yes SPAN-on-Drop Yes No SPAN-on-Latency Yes No SPAN with multiple destination ports *Software roadmap feature Yes (each destination port uses one SPAN session) No 63

64 High-Performance SPAN Scheduler Prioritizes Production Data Extra Fabric Bandwidth for SPAN Data SPAN Unified Crossbar Fabric Separate Buffer Pool for SPAN 64

65 SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 SPAN-on-Drop SPAN-On-Drop Tail-Drop Ingress Data Buffer SPAN Buffer Monitoring Station 65

66 SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 SPAN-on-Drop Copies the dropped packet to capture device Correlate the packet drop with application Works for unicast packets only Packet drops can be monitored on ingress only There is a dedicated buffer space for SPAN, so SPAN-on-Drop does not affect production traffic Supports both local SPAN and ERSPAN Configuration uses existing SPAN commands One SPAN-on-Drop session is supported Can have multiple source ports, and multiple destination ports Source cannot be FEX HIF port. But FEX fabric port is supported Source port(s) can be a part of a SPAN-on-Drop session, and a local SPAN session simultaneously 66

67 SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 SPAN-on-Drop The source interface is the ingress port for which we want to monitor drops. switch(config)# monitor session 1 type span-on-drop switch(config-span-on-drop)# source interface e3/1 switch(config-span-on-drop)# destination interface e3/2 switch(config)# monitor session 2 type span-on-drop-erspan switch(config-span-on-drop-erspan)# source interface e3/1 switch(config-span-on-drop-erspan)# destination ip

68 Latency Monitoring T0 T1 T0 Latency Monitoring: Min: 856ns Max: 1208ns Avg: 901ns T0 T1 T0 Latency Histogram: Latency Range: 800ns - 10usec Stats: Within the Range: x packets Out of the Range: y packets TimeStamp Packet FIFO Latency Per Port-Pair Data 68

69 Latency Latency Monitoring Monitoring - NX-OS 7.0(1)N1(1), Q1/2014 Latency Monitoring provides {min, average, max} latency between a specified port pair and also maintains latency histogram (accuracy in few nanosecs) By default instantaneous Latency Monitoring is enabled between pair of ports Latency Histogram can be enabled for specific port-pair to provide histogram instead of instantaneous mode Measures switch latency for each packet, no sampling required Fully implemented in HW, no CPU impact, no traffic impact 69

70 Instantaneous Mode Enabled by default on all pairs of ports. No configuration is required switch# show hardware profile latency monitor interface e1/23 interface e1/ Latency Statistics in nano seconds Ingress Port Egress Port Minimum Maximum Average Ethernet1/22 Ethernet1/

71 Custom Histogram Count the number of packets falling in a specific range of latency Example for port-pair e1/1 and e1/3: switch(config)# interface e1/3 switch(config-if)# packet latency low-latency 800 high-latency mode custom interface e1/1 switch# show hardware profile latency monitor histogram interface e1/3 interface e1/ Egress Interface : Ethernet1/3 Ingress Interface : Ethernet1/ All Latency Values are in nano seconds Range 800 <= Latency < Outside the first range counter

72 SPAN-on-Latency T0 T1 T0 Latency Monitoring: Min: 856ns Max: 1208ns Avg: 901ns T2 T3 T2 If Latency Threshold > 10 usec: SPAN to 1/64 TimeStamp Packet Monitoring Station 72

73 SPAN-on-Latency This feature replicates packets from the interface when latency exceeds a configured threshold Latency threshold is per-port This again helps to identify the victim of congestion Replicated traffic uses the SPAN buffer so it doesn't impact the production traffic One SPAN-on-Latency session is supported in hardware Uses similar SPAN CLI, with new session type span-on-latency ERSPAN-on-latency is also supported 73

74 SPAN-on-Latency Configuration The SPAN-on-Latency source port is the egress port on which we monitor the latency The SPAN-on-Latency session makes a copy of all high-latency packets egressing on this port, coming from any ingress port Sample configuration to SPAN packets if latency goes beyond 10us when egressing on e1/3: monitor session 1 type span-on-latency source interface Ethernet1/3 tx destination interface Ethernet1/4 Always Tx: packets egressing on 1/3 (any source) with latency >10us will be replicated to the SPAN dest 1/4 interface Ethernet1/3 hardware profile latency monitor threshold interface Ethernet1/4 switchport mode monitor 74

75 SPAN-on-Latency Source port can be an regular ethernet port, not a port-channel. Can be a port-channel member Source port cannot be FEX HIF port. But FEX fabric port is supported Multiple sources can be configured latency threshold is per SPAN-ondrop TX source port A SPAN-on-Latency source port cannot be in another SPAN session Destination is only a single ethernet port, not port-channel Only one destination port can be configured 75

76 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 76

77 Cisco Nexus 5600/6000 Multicast Features Summary IGMP snooping for IGMPv1, v2, and v3 PIM-SM PIM-SSM without vpc and PIM-SSM with vpc+ PIM-BiDir without vpc and PIM-BiDir with vpc+ PIM Policy MSDP Anycast RP with PIM (RFC 4610) Anycast RP with MSDP 8000 IGMP snooping entries 8000 mroutes with vpc, mroutes without vpc supported by software 77

78 Efficient Multicast Replication Optimized multicast replication Fabric replication and egress replication; one copy is replicated to egress UPC, where there is a receiver minimizing the traffic load on the switch fabric and eliminating the switch fabric congestion Line-rate multicast replication Egress UPC Ingress UPC Unified Crossbar Fabric Egress UPC 78

79 Multicast VOQ Unicast VOQ Multicast VOQ 8000 multicast VOQs to eliminate HOLB and help ensure high throughput Tracks the fan-out of the egress UPC; packets with different egress UPC fanouts are assigned to different VOQs so that it can be scheduled and served independently Ingress UPC Egress UPC Unified Crossbar Fabric Egress UPC Egress UPC

80 256,000 Entries Cisco Nexus 5600/6000 Key Forwarding Tables Host table: 256,000 -entry hashing table; actual capacity is slightly less than 256,000 Host table: Shared between MAC, ARP, and ND and /32 host route Host table default carving: 128,000 MAC, 128,000 IP hosts LPM table: 32,000 entries. Also known as summary routes Mroute table: 64,000 entries Host Table* LPM Table(32,000) Mroute Table(64,000)* MAC Region Summary Routes (S,G) IP Host Region (ARP/ND/Host route/(*,g)) * Hardware table size. Please check configuration limit for software scaling 80

81 Multicast VOQ Unicast VOQ MET Table 64,000 entries in MET table Currently supported: 8,000 vpc, 16,000 non-vpc Today the MET table is the limiting factor for multicast scale , 4, 5 1 Ingress UPC MET: Egress UPC 1 MET: 123 Switch Fabric MET: , 10 Egress UPC

82 Multicast VOQ Unicast VOQ Multicast Hashing over Port Channel N5600/6000 implements flow based hashing for multi-destination traffic, and it supports multicast load sharing over Port Channel with 5-tuple packet header. Traffic is replicated to all egress UPCs where Port Channel member resides Egress UPC runs hash calculation and one egress port is chosen to send out multicast packets. The UPC ASIC that is not supposed to send out packet will drop packet (Egress UPC 1 in the example) Egress UPC 1 Ingress UPC Port 3 Selected Port 1 Port 2 Unified Crossbar Fabric Egress UPC 2 Port 3 Selected Port 3 Port 4 82

83 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 83

84 Cisco Nexus 5600/6000 QoS Features Eight classes of service; 2 reserved for control traffic, 6 for data traffic Traffic classification DSCP, CoS, and ACL Strict Priority Queuing and DWRR DCBX 802.1Qaz Packet marking DSCP, CoS, and ECN Ingress and egress policing 4096 policers per ASIC No drop system class Flexible buffer management 84

85 Cisco Nexus 5600/6000 QoS Processing Flow Ingress UPC Trust CoS/DSCP L2/L3/L4 info with ACL If Buffer Usage Crosses Threshold: Tail drop for drop class Assert pause signal to MAC for no-drop system class VOQs for Unicast (8 per egress port) MAC MAC Traffic Classification Ingress Cos/DSCP Marking Ingress Policing PAUSE ON/OFF Signal ECN Marking MTU Checking Truncate or Drop Packets if MTU is Violated Egress Policing Per-class Buffer Usage Monitoring Egress Scheduling Multicast Queues Egress Queues Unicast Crossbar Fabric Egress UPC Strict Priority + DWRR Scheduling Multicast 85

86 Multicast VOQ Unicast VOQ Increased Packet Buffer 25MB packet buffer is shared by every three 40GE ports or twelve 10GE ports. Nexus 5672: Nexus 56128: Nexus 6004: 150MB of total buffer 300MB of total buffer 800MB of total buffer Ingress UPC Egress UPC 16MB Unified Crossbar Fabric 9MB 86

87 Ingress Buffer - Flexible Buffer Management Shared buffer is good for burst absorption. Dedicated buffer is good for predictable performance for each port. N5600/6000 buffer management is flexible: dedicated plus shared. Long-distance FCoE, video editing (i.e., AVID), Big Data, and distributed storage UPC Ingress Buffer (16MB) Port 1 Dedicated SPAN Port 2 Dedicated Control Shared Packet Buffer ~14MB Port 3 Dedicated 87

88 Tune Shared vs Dedicated Buffer Allocation at Ingress queue-limit under network-qos policy specifies the dedicated buffer for each port and each class. Without queue-limit each class of service will get 100 KB of dedicated buffer. The size of dedicated buffer can be different for different classes of service. The policy applies to all ports in the chassis. Total ingress buffer minus the dedicated buffer and buffer for control and SPAN will be in the shared buffer pool. The following example sets the dedicated buffer for class-default to be 400 KB for all ports. switch(config)# policy-map type network-qos Policy-buffer switch(config-pmap-nq)# class type network-qos class-default switch(config-pmap-nq-c)# queue-limit bytes switch(config-pmap-nq-c)# system qos switch(config-sys-qos)# service-policy type network-qos Policy-buffer 88

89 Egress Buffer Design 9-MB buffer is reserved for egress traffic Unicast traffic can be buffered at egress and ingress. Multicast is buffered at egress in case of interface oversubscription. UPC Egress Buffer (9MB) Unicast Buffer Dedicated per port Multicast Buffer Shared ~6MB 89

90 Unicast Flow Control and Buffering Unicast is queued at egress buffer pool first and then ingress buffer after egress queue is full. Take advantage of ingress buffer from multiple port or ASIC for unicast burst absorption. Ensure fairness among multiple ingress ports 90

91 Identify Packet Drops Due to Congestion switch# show interface ethernet 1/7 Ethernet1/1 is down (SFP not inserted) Dedicated Interface <snip> RX <snip> 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble input discard 0 Rx pause TX 0 unicast packets 0 multicast packets 0 broadcast packets 0 output packets 0 bytes 0 jumbo packets 0 output error 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause Broadcast/Multicast/Unknown unicast flooding drop at egress Unicast drop at ingress 91 switch# show queuing interface ethernet 1/7 Ethernet1/1 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 100 RX Queuing qos-group 0 Dedicated buffer for this class q-size: , HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: 0 Statistics: Pkts received over the port : Ucast pkts sent to the cross-bar : Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 0 Pkts discarded on ingress : Per-priority-pause status : Rx (Inactive), Tx (Inactive) Ingress drop for each class

92 Identify Congested Egress Port with VOQ Counters I X I A I X I A 1/7 1/10 Ingress UPC 3 Ingress UPC 4 Unified Crossbar Fabric Egress UPC 3 1/8 I X I A Using show interface CLI, we noticed drops on ingress ports How do we know which egress port is affected? 92

93 Step 1: Map Interface to UPC Number N6004# show hardware internal bigsur all-ports Bigsur Port Info: Port asic inst inst name idx slot asic eport logi flag adm opr if_index diag ucver sup b3 en dn pass 0.00 sup b3 en dn pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 dis dn 1a pass gb1/ b3 dis dn 1a pass gb1/ p 6 b3 en up 1a pass gb1/ p 7 b3 en up 1a pass gb1/ p 8 b3 en up 1a pass gb1/ p 9 b3 en up 1a pass gb1/ b3 dis dn 1a07a000 pass gb1/ p 11 b3 en up 1a07b000 pass 0.00 Done. 93

94 Step 2: Identify Congested Egress Port with VOQ Counters N6004# show platform software qd info counters voq asic-num port TRANSMIT TAIL DROP HEAD DROP Eth1/8 QUEUE Congested egress port and class of service Ingress UPC number N6004# show platform software qd info counters voq asic-num port TRANSMIT TAIL DROP HEAD DROP Eth1/8 QUEUE

95 Nexus 5600/6000 Architecture - Agenda Overview Fabric Extender Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 95

96 Nexus 5600/6000 Applications Compact Aggregation Nexus 7000 CORE Nexus 7000 Nexus 6004 L3 AGG. Nexus 6004/ Nexus 5600 vpc to Nexus 7000 or 6004 ACCESS Nexus 2000 FEX 96

97 Nexus 5600/6000 Applications Large Scale Fabric (Layer 2 or Layer 3) Nexus 7000 CORE L3 Nexus 6004 SPINE Nexus 6004/ Nexus FabricPath LEAF 97

98 Nexus 5600/6000 Applications High Performance Computing (HPC) Nexus 6004 CORE/AGG. L3/L2 Fabric Nexus 3548 Nexus 5600/6004 ACCESS 98

99 Nexus 6004X Chassis Target Q3 CY14 Nexus 6004X has 8 slots (similar to the 6004EF chassis) Nexus 6004X chassis supports VXLAN functionality Provides Flexible Ordering Options mix/match 10/40G LEMs Supports new 12 x 40G LEM VXLAN capable (backward compatible with Nexus 6004EF chassis) VXLAN capable LEM will work in non-vxlan mode in 6004EF chassis Same software as on Nexus 6004 Supports all the optics, FEX models as on Nexus

100 Nexus 5600/6000 Key Takeaways Rich Feature set Performance Scalability Innovations STP/vPC/FabricPath/DFA, FEX, L3, VXLAN, Adapter-FEX/VM-FEX Line rate L2 and L3 with low latency 256K MAC / 128K ARP / 32K LPM routes, 16K bridge domains with segment ID Intelligent SPAN, Buffer monitoring, Latency monitoring 100

101 Nexus 2000/5600/6000 More Information

102 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online 102

103 Continue Your Education Demos in the Cisco Campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings 103

104

105