Cisco Nexus 5500, 5600, 6000 with Fabric Extender 2000 Switch Architecture

Transcription

1

2 Cisco Nexus 5500, 5600, 6000 with Fabric Extender 2000 Switch Architecture Nicolas Delecroix Technical Marketing Engineer Data Center Group Nexus Business Unit

3 Nexus 5000 History Nexus 5020 Customers 56-Port, 2RU Nexus Port, 1RU Nexus 5596T 10Gbase-T, 2RU Nexus Port, 1RU Nexus Port, 2RU 3

4 Introducing Cisco Nexus 5600/6000 Nexus port 10G, 4 port 40G, 1RU Nexus 6004 Nexus 5020 Customers 56-Port, 2RU Nexus Port, 1RU Nexus 5596T 10Gbase-T, 2RU 96 port 40G, 4RU Nexus Port, 1RU Nexus Port, 2RU 4

5 Introducing Cisco Nexus 5600/6000 Cisco Nexus 5600/6000 Series Nexus port 10G, 4 port 40G, 1RU Nexus port 10G, 6 port 40G, 1RU Nexus Nexus port 10G, 8 port 40G, 2RU Nexus 5020 Customers 56-Port, 2RU Nexus Port, 1RU Nexus 5596T 10Gbase-T, 2RU 96 port 40G, 4RU Nexus Port, 1RU Nexus Port, 2RU 5

6 Introducing Cisco Nexus 5600/6000 Nexus 6004 (40G Optimized) 40G Nexus 6000 (40G Optimized) Nexus 5010/5020 Nexus 5500 (10G Optimized) Nexus 6001 (10G Optimized) 10G Nexus 5600 (10G Optimized) 1 st Generation nd Generation rd Generation

7 Cisco Nexus 5500, 5600, 6000 with Fabric Extender 2000 Switch Architecture Nicolas Delecroix Technical Marketing Engineer Data Center Group Nexus Business Unit

8 Nexus 5600/6000 Architecture Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 8

9 Introducing Cisco Nexus 5600/6000 High Performance Nexus 5672 Nexus Line rate L2 and L3, with all ports, all features, all frame sizes 1-microsecond latency with all frame sizes 40GE flow 40GE FCoE Nexus MB buffer per 12xSFP / 3xQSFP

10 Introducing Cisco Nexus 5600/6000 High Scalability Nexus 5672 Nexus : 48x10G + 6x40G 56128: 96x10G + 8x40G 6004: 96x40G / 384x10G Up to 256,000 MAC Up to 128,000 ARP 32,000 LPM Nexus 6004

11 Introducing Cisco Nexus 5600/6000 Feature-Rich Nexus 5672 L2 and L3 FEX Nexus vpc, FabricPath FabricPath with segment ID Adapter-FEX / VM-FEX Nexus 6004

12 Introducing Cisco Nexus 5600/6000 Nexus 5672 Visibility and Analytics Line-rate SPAN, 16 sessions (Hardware capable of 31 sessions) Sampled NetFlow Nexus Micro-burst and buffer monitoring Latency monitoring Nexus 6004 Conditional SPAN: SPAN on drop, SPAN on higher latency

13 Cisco Nexus 6004 Chassis 12 QSFP ports Expansion Module 4RU 30 in.

14 Cisco Nexus 6004 Chassis Power Supply and Fans Six power supply slots; a minimum of three is required. Supports both and redundancy. Each power supply is rated 1100W, VAC. The chassis has four fan trays. A minimum of three is required. Power Supply Grid Redundancy or Redundancy Console Mgmt0 USB Fan Module 3+1 Redundancy 14

15 Unified Port Line-Card Expansion Module For Nexus SFP+ unified ports expansion module Planned 1HCY14 2/4/8/FC port 1G/10GE and FCoE For Nexus 6004 only SFP+ port allows support for more variety of optical transceiver 15

16 Nexus 5672 Chassis Port-Side View Built with same ASIC family as Nexus 6004 Available March CY14 16 Unified Ports provide 2/4/8G FC, 10/40G Ethernet/FCoE L2 and L3 VXLAN 48 fixed 1/10G SFP+ interfaces 6 fixed 40G QSFP interfaces 16

17 Cisco Nexus 5672 Chassis Power Supply and Fans Power Supply 1+1 Redundancy Fan Module 2+1 Redundancy Console Mgmt0 USB 17

18 Nexus Chassis Port-Side View Built with same ASIC family as Nexus 6004 Available March CY14 48 Fixed 10G SFP+ Ports, 4 fixed 40G QSFP+ ports 2 Expansion Slots Expansion module provides 24 Unified ports 10G Ethernet/FCoE or 2/4/8G FC, and 2 ports 40G Ethernet/FCoE 18

19 Cisco Nexus Chassis Power Supply and Fans Power Supply 2+2 Redundancy Console Mgmt0 USB Fan Module 3+1 Redundancy 19

20 Cisco Nexus 5600/6000 Airflow The units have a port side and a fan side. No front or back The units support both port side exhaust and port side intake Port side exhaust (used for in server rack airflow alignment) Port side intake (used for network rack airflow alignment) Different power supply and fan modules are required for different airflow directions

21 N5600/6000 and N5500 Comparison The table lists the Nexus 5600/6000 hardware features and scalability numbers. Please check software roadmap for availability Feature N5600/6000 N5548/N5596 Latency ~1 us 1.8us L2 Throughput Line rate Line rate L3 Throughput Line rate 160Gbps Native 40GE support Yes No (4xQSFP module on the roadmap) 1GE support Yes Yes Unified Port Future hardware Yes FEX Scaling 48 (6004 L2), 24 (6004 L3, 5600) 24 with L2, 16 with L3 MAC Table 256K (shared with ARP/ND) 32K IPv4 Unicast routes 32K 16K IPv4 ARP/host route entries 128K 16K (shared with IPv6 ND and mroute) IPv4 multicast routes 32K 8K (shared with ARP) 21

22 N5600/6000 and N5500 Comparison (cont.) The table lists the Nexus 5600/6000 hardware features and scalability numbers. Please check software roadmap for availability Feature N5600/6000 N5548/N5596 IPv6 unicast routes 8K (shared with IPv4) 8K (shared with IPv4) IPv6 ND entry/host route entries 85K (shared with MAC and ARP) 8K (shared with mroute and ARP) IPv6 multicast routes 32K VLAN 4K 4K Bridge Domain 16K 4K VRF 4K 1K Segment ID Yes No ACL 4K 4K 8K (shared with ARP, ND. Limit is with /64 routes) L4ops for ACL 24 total (16 for UDP/TCP, 8 for TCP flag) 8 for UDP and 8 for TCP Packet Buffer 25MB per 3xQSFP (or 12x10G) 2.1MB per 10G port 640KB per 10G port Buffer management Dedicated plus shared Dedicated per port 22

23 N5600/6000 and N5500 Comparison (cont.) The table lists the Nexus 5600/6000 hardware features and scalability numbers. Please check software roadmap for availability Feature N5600/6000 N5548/N5596 SPAN 31 bi-directional (16 at FCS) 4 bi-directional ERSPAN source session 16 4 ERSPAN destination session 16 No Line rate SPAN Yes No Prioritize data over SPAN Yes (through scheduler) Yes (SPAN policing) Conditional SPAN Yes No Multicast Replication Fabric and egress replication Fabric PIM-BiDir Yes No IP based forwarding for IGMP snooping Yes No 23

24 48xFEX Nexus 2000 Fabric Extender Scalability 48 FEX per 6004 in L2, 24 in L3 24 FEX in L2 and L3 for 5600 Support for all different types of FEX including HP, Fujitsu, Dell, IBM FEX for blade servers Cisco Nexus 5600/6000 Cisco Nexus 2000 FEX + 24

25 Nexus 2000 Fabric Extender Portfolio Nexus 5600/6000 supported 100M/1G Fabric Extenders Nexus 2224TP Nexus 2248TP Nexus 2248TP-E Form Factor 1 RU 1 RU 1 RU Uplink Ports 2 x 10G SFP+ 4 x 10G SFP+ 4 x 10G SFP+ Host Facing Ports 24 x 100/1000BASE-T RJ45 48 x 100/1000BASE-T RJ45 48 x 100/1000BASE-T RJ45 Oversubscription 1 : : : 1.2 FCoE N/A N/A N/A Shared Buffer No No Yes 25

26 Nexus 2000 Fabric Extender Portfolio Nexus 5600/6000 supported 1/10G Fabric Extenders Nexus 2232PP Nexus 2248PQ Nexus 2232TM Nexus 2232TM-E Form Factor 1 RU 1 RU 1 RU 1 RU Uplink Ports 8 x 10G SFP+ 4 x 40G QSFP+ 8 x 10G SFP+ 8 x 10G SFP+ Host Facing Ports 32 x SFP/SFP+ (1/10G) 48 x SFP/SFP+ (1/10G) 32 x 1/10GBASE- T RJ45 32 x 1/10GBASE-T RJ45 Oversubscription 1 : 4 1 : 3 1 : 4 1 : 4 FCoE Yes Yes No Yes (30m / Cat6a/7) Shared Buffer No Yes No No 26

27 Nexus 5600/6000 Architecture - Agenda Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 27

28 N6004 Performance Fully Loaded Switch

29 N6004 Performance Spirent Traffic Generator

30 N x 40G Unicast Throughput Throughput (% of line rate) RFC 2544 Unicast L2 Port Pair RFC 2544 Unicast L3 Port Pair Packet Size (Bytes) Spirent third party performance report for Nexus 6004

31 N x 40G Multicast Throughput Throughput (% of line rate) RFC 3918 Mcast L2 One to Many Accumulated RFC 3918 Mcast L3 One to Many Accumulated RFC 3918 Mcast L2 One to Many Distributed RFC 3918 Mcast L3 One to Many Distributed Packet Size (Bytes) Spirent third party performance report for Nexus 6004

32 N x 10G Unicast 100% Load Latency Average Latency 1 (microseconds) RFC 2544 Unicast L2 Port Pair RFC 2544 Unicast L3 Port Pair Packet Size (Bytes)

33 N x 10G Multicast 100% Load Latency Average Latency (microseconds) RFC 3918 Mcast L2 One-to-Many Accumulated RFC 3918 Mcast L3 One-to-Many Accumulated RFC 3918 Mcast L2 One-to-Many Distributed RFC 3918 Mcast L2 One-to-Many Distributed

34 N x 10G Unicast and Multicast 100% Load Jitter Average Jitter (nanoseconds) Unicast L2 Jitter (ns) Unicast L3 Jitter (ns) Multicast L2 Jitter (ns) Multicast L3 Jitter (ns) Packet Size (Bytes)

35 Nexus 5600/6000 Architecture - Agenda Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 35

36 Cisco Nexus 5672 Internal Architecture Switch Fabric Unified Fabric Controller (UFC) UPC 1 UPC 2 UPC 3 UPC 4 UPC 5 UPC 6 Supervisor UPC-0 48xSFP+ 6xQSFP+ CPU 36

37 Cisco Nexus Internal Architecture 24x SFP+ Slot-2 2x QSFP+ 24x SFP+ Slot-3 2x QSFP+ UPC 7 UPC 8 UPC 9 UPC 10 UPC 11 UPC 12 Supervisor Unified Fabric Controller (UFC) UPC-0 UPC 1 UPC 2 UPC 3 UPC 4 UPC 5 UPC 6 CPU 24x SFP+ 24x SFP+ 2x QSFP+ 2x QSFP+

38 Cisco Nexus 6004 Internal Architecture Switch Fabric UFC-1 UFC-2 UFC-3 UFC-4 Supervisor UPC 1 UPC 2 UPC 16 UPC 17 UPC 32 UPC-0 96xQSFP CPU 38

39 Unified Port Controller - UPC ASIC Multimode MAC; built-in PHY for 1, 10, and 40 GE Packet parsing and rewriting Lookup engine and access control: L2, L3, FabricPath, ACL, FCoE, and policing Buffering and queuing: Buffer management, PFC for lossless traffic, queuing (Strict Priority Queuing and DWRR), and packet replication (SPAN and multicast) Extra fabric bandwidth for SPAN and multi-destination traffic Front Panel Ports Twelve 10 GE UPC Fabric Connections 224Gbps 448Gbps Three 40 GE UPC 224Gbps 448Gbps 39

40 Cisco Nexus 5600/6000 Packet Processing Flow Ingress Pipeline 1, 10, and 40 GE MAC Parser Security ACL QoS ACL SPAN Filter Policing/Stats Storm Control VLAN Table MAC Table ARP/ND Table IPv4/IPv6 RoutingTable FabricPath FC routing Table Forwarding Lookup ACL Ingress Policing Ingress SPAN Replication Packet Rewrite Ingress UPC Buffer Allocation Buffer Accounting PAUSE Signal Buffer Management Buffer Unicast VOQ (8 per egress port) 8000 Multicast VOQs Unified Crossbar Fabric 40

41 Unified Crossbar Fabric Nexus 5600/6000 Packet Processing Flow Egress Pipeline Egress UPC Unicast buffer Multicast congestion management Unicast Queue Buffer Management Scheduling Multicast Replication Egress ACL Egress Policing* Editing MAC Buffer Multicast Queue TX SPAN Replication *Egress policing is not supported with current software To Ingress Buffer Management 41

42 256,000 Entries Cisco Nexus 5600/6000 Key Forwarding Tables Host table: 256,000 -entry hashing table; actual capacity is slightly less than 256,000 Host table: Shared between MAC, ARP, and ND and /32 host route Host table default carving: 128,000 MAC, 128,000 IP hosts LPM table: 32,000 entries. Also known as summary routes Mroute table: 64,000 entries Host Table* LPM Table(32,000) Mroute Table(64,000)* MAC Region Summary Routes (S,G) IP Host Region (ARP/ND/Host route/(*,g)) * Hardware table size. Please check configuration limit for software scaling 42

43 256,000 Entries Cisco Nexus 5600/6000 Host Table Carving In virtualized environments, there is a larger number of MAC addresses compared to IP addresses. One gateway (IP) with multiple VMs behind (MAC). MAC/ARP Hardware Resource Carving allows to change the size of MAC and IP regions to allocate more space for the MAC region. Host Table MAC Region IP Host Region (ARP & ND & /32 Host route)

44 Cisco Nexus 5600/6000 Carving Profiles Use of pre-defined carving profiles. HRT = Host Route Table = IP host region. STM = Station Table Management = MAC region. Template Profile hrt-128-stm-128 Description HRT size: 128k, STM size: 128k (default profile) hrt-96-stm-160 HRT size: 96k, STM size: 160k hrt-64-stm-192 hrt-32-stm-224 HRT size: 64k, STM size: 192k HRT size: 32k, STM size: 224k

45 Cisco Nexus 5600/6000 MAC/ARP Resource Carving CLI Specify the resource template to use: switch(config)# hardware profile route resource service-template template-name Need to save the config and reload the switch to be applied. Show commands: show hardware profile route resource template show hardware profile route resource template default show running-config hardware profile route resource template show startup-config hardware profile route resource template

46 Nexus 5600/6000 Architecture - Agenda Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 46

47 QSFP-40G-SR4 40G-SR4 40G-SR4 MPO-12 MPO-12 40G-SR4 10G-SR 10G-SR 10G-SR 10G-SR Support 100M with OM3 MMF and 150M with OM4 MMF MPO-12 LC Breakout 47

48 QSFP-40G-CSR4 40G-CSR4 MPO-12 MPO-12 40G-SR4/CSR4 Supports 300M with OM3 MMF and 400M with OM4 MMF Compatible with Cisco QSFP- 40G-SR4 Compatible with 10G SFP-SR 40G-CSR4 10G-SR 10G-SR 10G-SR 10G-SR MPO-12 LC Breakout 48

49 Multimode Ribbon Fiber 40GE For QSFP-40G-SR4 and QSFP-40G-CSR4 12-Fiber MPO Connector MPO/MTP connector with 12 pins Use 4 fiber pairs 4 TX and 4 RX allows for 40 GE 4 unused fibers in the center 49

50 QSFP-40G-LR4 QSFP-40G-LR4 can only be connected to each other. 10KM with Single mode fiber. 4x ROSA Multiplexer Host Card TIA 4x TOSA LC LC IEEE Standard 40GBASE-LR4 Can t connect to 10Gbase-LR 50

51 QSFP 40G Bidirectional Transceiver 40G deployment with 10G cabling Cisco s innovation has created a new QSFP pluggable 40Gbit/s Bidirectional (BiDi) Transceiver that works with existing infrastructure: Multi-mode fiber Using the existing 10Gbit/s Multimode infrastructure QSFP-40G-SR-BD has dual LC Connector 40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and received simultaneously on two wavelengths 51

52 QSFP-40G-SR-BD NX-OS 7.0(1)N1(1), Q1/ G-BD Multimode LC Duplex 40G-BD Supports 100m with OM3 with MMF and 150m with OM4 (OM4+) MMF LC duplex connectors Wavelength: ~850nm and ~900nm Pull-Tab Color: grey 40G-BD Optical DeMux 10G-SR 10G-SR 10G-SR 10G-SR Multimode LC Duplex 52

53 Physical Connection Choices 40G QSFP+ QSFP-LR4 QSFP-LR4 40G QSFP+ 40G QSFP+ QSFP-SR-BD QSFP-SR-BD 40G QSFP+ 40G QSFP+ QSFP-SR4/CSR4 QSFP-SR4/CSR4 40G QSFP+ 40G QSFP+ 1M, 3M, 5M Passive 7M, 10M Active 40G QSFP+ 40G QSFP+ 40G QSFP+ QSFP-SR4/CSR4 QSFP-4x10G 10G SFP+ 10G SFP+ 53

54 FET-40G Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004 Supported on Cisco Nexus 5600/6004 and Nexus 2248PQ-10G Interoperable with FET-10G Support for 100m distance with OM3 Cisco Nexus 5600/6000 Cisco Nexus 5600/6000 FET-40G FET-40G FET-40G Cisco Nexus 2248PQ-10G FET-10G Cisco Nexus 2232PP and 2232TM-E 2232TM Cisco Nexus 2248TP-E 2248TP 54

55 QSA-QSFP to SFP Adapter Support from 6.0(2)N2(1) Convert one QSFP port to one SFP+ port. Provides 1G support on Nexus 6004 Long reach connection between N6004 and 10G device Following 1G/10G transceivers are qualified with QSA in release-6.0(2)n2(1) 10G-LR 1G GLC-T, 1G SX, LX Requires a mode change on QSFP port to 4x10G Only first 10G interface can be used. Remaining three 10G shown as Link Not Connected QSFP+ Adapter 1G SFP or 10G SFP+ Transceiver 55

56 Nexus 5600/6000 Architecture - Agenda Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 56

57 ACL Types and Features Security ACL MAC, IPv4, and IPv6 ACLs PACL: ACL enabled under L2 interface VACL: ACL enabled for L2 VLAN traffic RACL: ACL enabled for routed traffic RBACL: Role-based ACL for CTS (roadmap feature) ACL for QoS classification and Policing Policy Based Routing (PBR) User-configured ACL to redirect traffic ACL for SPAN/ERSPAN (roadmap feature) ACL for control traffic To redirect control traffic to CPU, not user-configurable CoPP 57

58 4096 Entries ACL Scaling and TCAM Partition 4096 Access Control Entries per UPC are organized in blocks of 64 ACE entries. TCAM Carving provides the possibility to change the size of each region. Each IPv6 ACL (without port range) requires two TCAM entries. Default TCAM Partition VACL(1024) IFACL(1152) QoS(448) RBACL(1024) SPAN(64) Control Traffic(256) N6000-TME1# sh platform afm info tcam 0 region ifacl ifacl tcam TCAM configuration for asic id 0: [ vacl tcam]: range [ifacl tcam]: range * [ qos tcam]: range [rbacl tcam]: range [ span tcam]: range [ sup tcam]: range TCAM [ifacl tcam]: [v:1, size:1152, start:1024 end:2175] In use tcam entries: 24 TCAM Region , Usage 58

59 TCAM Carving TCAM Carving allows to change the size of each region Create a template: switch(config)# hardware profile tcam resource template t1 switch(config-tcam-templ)# Set region sizes: switch(config-tcam-templ)# ifacl? < > Enter size of ifacl region (in 64 entry increments)

60 TCAM Carving Similarly other region sizes can be changed: switch(config-tcam-templ)#? ifacl Configure size of ifacl region no Change size to default value qos Configure size of qos region rbacl Configure size of rbacl region span Configure size of span region vacl Configure size of vacl region This command displays the configuration for a template along with the current usage: switch(config)# show hardware profile tcam resource template name? WORD Select name of a template

61 TCAM Carving switch(config-tcam-templ)# show hardware profile tcam resource template name t Template name: t1 Current state: Created Region Size-allocated Current-size Current-usage Available/free Vacl Ifacl Rbacl Qos Span Sup

62 TCAM Carving Commit a template: switch(config)# hardware profile tcam resource service-template t1 Details of the t1 template you are trying to commit are as follows: Template name: t1 Current state: Committed (Startup/Running) Region Features Size-allocated Current-size Current-usage Available/free Vacl Vacl [ ] Template committed. Please do the following for the template to be applied: 1> Save running config : "copy running-config startup-config" 2> Reboot the switch : "reload

63 ACL Logging Logs the packets that hit ACL rules Supported on all flavors of ACL: PACL, RACL, VACL & RBACL ACL logging feature allows the logging of the packets hitting IPv4 ACL only. ACL logging is not supported for IPv6 and MAC ACLs. Supported on all interfaces including FEX HIF interfaces and management. Logs only packets that hit the deny rule. switch(config)# ip access-list example switch(config-acl)# permit tcp any any log

64 ACL Logging Message and CLI Log is displayed on a flow basis. Flow is identified using the combination of IP source address, destination address, L4 protocol and the L4 source/destination ports ACLLOG-3-ACLLOG_FLOW_INTERVAL: Source IP: , Destination IP: , Source Port: 58768, Destination Port: 22, Source Interface: Ethernet1/1, Protocol: "TCP"(6), Hit-count = 1 switch# show log ip access-list cache Source IP Destination IP S-Port D-Port Interface Protocol Hits Ethernet2/11 (1)ICMP Ethernet2/10 (1)ICMP 5 Number of cache entries: switch#

65 ACL Logging Message Conditions The ACL log message will be generated on the following conditions: When a new flow is created (INFO message) When packet threshold is reached for a flow (WARNING message) At the end of a periodic interval (default 5 mins) with the info about how many packets hit the flow. (INFO message configurable)

66 Nexus 5600/6000 Architecture - Agenda Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 66

67 Switched Port Analyzer (SPAN) Switch Host-A Host-B Monitor Source Monitor Destination Sniffer Device 67

68 Encapsulated Remote SPAN (ERSPAN) Host-B GRE Encapsulated monitored traffic Host-A IP Cloud Monitor Source Monitor Destination Sniffer Device 68

69 Cisco Nexus 5600/6000 SPAN Hardware Features SPAN Features Cisco Nexus 5600/6000 Total SPAN sessions 31 (16 with current software) Local SPAN sessions 31 (16 with current software) ERSPAN sessions 16 Prioritize data over SPAN Yes (through scheduling) Line-rate SPAN throughput Yes ERSPAN destination session* Yes ERSPAN with 1588 PTP timestamp* Yes MTU Truncated SPAN and ERSPAN Yes ACL filter for SPAN and ERSPAN* Yes SPAN on drop Yes SPAN on high latency Yes SPAN with multiple destination ports* Yes (Each destination port uses one SPAN session.) *Software roadmap features 69

70 Ingress SPAN Packet Flow Data is replicated at ingress port ASIC-Unified Port Controller (UPC). SPAN packets are queued at the SPAN destination port VOQ. Ingress Interface (RX SPAN source) Egress Interface Data SPAN Unicast VOQ Multicast VOQ Data SPAN Unified Fabric Controller Data SPAN Destination SPAN 70

71 Egress SPAN Packet Flow SPAN copy is made at egress pipe of the TX SPAN source port. SPAN packets are looped back to ingress pipe of UPC and sent to switch fabric. Ingress Interface Egress Interface (TX SPAN source) Unicast VOQ Data Data Unicast VOQ SPAN Data Multicast VOQ Unified Crossbar Fabric SPAN SPAN Destination 71

72 High-Performance SPAN No penalty to production data traffic with SPAN Extra bandwidth capacity built in for SPAN traffic First product in the market to support 16 line-rate SPAN sessions Scheduler Prioritizes Production Data Extra Fabric Bandwidth for SPAN 448 Gbps 120-Gbps Data 120-Gbps Data Data SPAN 224 Gbps Unified Crossbar Fabric 448 Gbps Separate Buffer Pool for SPAN 72

73 Line-Rate Multicast with Line-Rate SPAN 240-Gbps replication capacity for each UPC 120-Gbps for multicast; 120-Gbps for SPAN 240-Gbps Replication per UPC Multicast SPAN I X I A 40GE Multicast Source Ingress UPC Unified Crossbar Fabric Egress UPC Egress UPC Receiver 40GE Receiver 40GE Receiver 40GE SPAN Destination SPAN SPAN I X I A I X I A 73

74 SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 SPAN-On-Drop Tail-Drop Ingress Data Buffer SPAN Buffer Monitoring Station

75 SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 Copies the dropped packet to capture device Correlate the packet drop with application Works for unicast packets only Packet drops can be monitored on ingress only There is a dedicated buffer space for SPAN, so SPAN on Drop does not affect production traffic Supports both local SPAN and ERSPAN Configuration uses existing SPAN commands One SPAN on Drop session is supported

76 SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 The source interface is the ingress port for which we want to monitor drops. switch(config-if)# monitor session 1 type span-on-drop switch(config-span-on-drop)# source interface ethernet 3/1/1 switch(config-span-on-drop)# destination interface ethernet 3/1/2 switch(config-span-on-drop)# no shut switch(config)# monitor session 2 type span-on-drop-erspan switch(config-span-on-drop-erspan)# source interface ethernet 3/1/1 switch(config-span-on-drop-erspan)# destination ip

77 Latency Monitoring - NX-OS 7.0(1)N1(1), Q1/2014 T0 T1 T0 Latency Monitoring: Min: 856ns Max: 1208ns Avg: 901ns T0 T1 T0 Latency Histogram: Latency Range: 800ns - 10usec Stats: Within the Range: x packets Out of the Range: y packets TimeStamp Packet FIFO Latency Per Port-Pair Data

78 Latency Monitoring - NX-OS 7.0(1)N1(1), Q1/2014 Instantaneous latency and histogram latency between a pair of ports (ingress and egress port pair) Feature provides {min, average, max} latency between specified port pair and also maintains latency histogram (accuracy in few nanosecs) By default instantaneous Latency Monitoring is enabled between pair of ports Latency Histogram can be enabled for specific port-pair to provide histogram Either latency monitoring or histogram can be enabled between same port-pair Instantaneous Latency Monitoring: switch# show hardware profile latency monitor interface e1/23 interface e1/ » Latency Statistics in nano seconds Ingress Port Egress Port Minimum Maximum Average Ethernet1/22 Ethernet1/

79 Latency Histogram Config - NX-OS 7.0(1)N1(1), Q1/2014 Latency Histogram configuration to change default latency monitoring between port-pair e1/1 and e1/3: switch(config)# interface e1/3 switch(config-if)# packet latency low-latency 800 high-latency mode custom interface e1/1 switch# show hardware profile latency monitor histogram interface e1/3 interface e1/ Egress Interface : Ethernet1/3 Ingress Interface : Ethernet1/ All Latency Values are in nano seconds Range 800 <= Latency < Outside the first range counter

80 SPAN on Latency - NX-OS 7.0(1)N1(1), Q1/2014 T0 T1 T0 Latency Monitoring: Min: 856ns Max: 1208ns Avg: 901ns T2 T3 T2 If Latency Threshold > 10 usec: SPAN to 1/64 TimeStamp Packet Monitoring Station

81 SPAN on Latency Configuration - NX-OS 7.0(1)N1(1), Q1/2014 The SPAN-on-Latency source port is the egress port on which we monitor the latency The SPAN-on-Latency session makes a copy of all high-latency packets egressing on this port, coming from any ingress port Sample configuration to SPAN packets if latency goes beyond 10us when egressing on e1/3: monitor session 1 type span-on-latency source interface Ethernet1/3 tx destination Always tx direction interface Ethernet1/4 interface Ethernet1/3 hardware profile latency monitor threshold interface Ethernet1/4 switchport mode monitor

82 Nexus 5600/6000 Architecture - Agenda Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 82

83 Cisco Nexus 5600/6000 Multicast Features Summary As of Release 6.0(2)N2(3) IGMP snooping for IGMPv1, v2, and v3 PIM-SM PIM-SSM without vpc and PIM-SSM with vpc+ PIM-BiDir without vpc and PIM-BiDir with vpc+ PIM Policy MSDP Anycast RP with PIM (RFC 4610) Anycast RP with MSDP 8000 IGMP snooping entries 8000 mroutes with vpc, mroutes without vpc supported by software 83

84 Efficient Multicast Replication Optimized multicast replication Fabric replication and egress replication; one copy is replicated to egress UPC, where there is a receiver minimizing the traffic load on the switch fabric and eliminating the switch fabric congestion Line-rate multicast replication Egress UPC Ingress UPC Unified Crossbar Fabric Egress UPC 84

85 Multicast VOQ Unicast VOQ Multicast VOQ 8000 multicast VOQs to eliminate HOLB and help ensure high throughput Tracks the fan-out of the egress UPC; packets with different egress UPC fanouts are assigned to different VOQs so that it can be scheduled and served independently Ingress UPC Egress UPC Unified Crossbar Fabric Egress UPC Egress UPC

86 Multicast VOQ Unicast VOQ Multicast Hashing over Port Channel N5600/6000 implements flow based hashing for multi-destination traffic, and it supports multicast load sharing over Port Channel with 5-tuple packet header. Traffic is replicated to all egress UPCs where Port Channel member resides Egress UPC runs hash calculation and one egress port is chosen to send out multicast packets. The UPC ASIC that is not supposed to send out packet will drop packet (Egress UPC 1 in the example) Egress UPC 1 Ingress UPC Port 3 Selected Port 1 Port 2 Unified Crossbar Fabric Egress UPC 2 Port 3 Selected Port 3 Port 4 86

87 Nexus 5600/6000 Architecture - Agenda Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 87

88 Cisco Nexus 5600/6000 QoS Features Eight classes of service; 2 reserved for control traffic, 6 for data traffic Traffic classification DSCP, CoS, and ACL Strict Priority Queuing and DWRR DCBX 802.1Qaz Packet marking DSCP, CoS, and ECN Ingress and egress policing 4096 policers per ASIC No drop system class Flexible buffer management 88

89 Cisco Nexus 5600/6000 QoS Processing Flow Trust CoS/DSCP L2/L3/L4 info with ACL Ingress UPC If Buffer Usage Crosses Threshold: Tail drop for drop class Assert pause signal to MAC for no-drop system class VOQs for Unicast (8 per egress port) MAC MAC Traffic Classification Ingress Cos/DSCP Marking Ingress Policing PAUSE ON/OFF Signal ECN Marking MTU Checking Truncate or Drop Packets if MTU is Violated Egress Policing Per-class Buffer Usage Monitoring Egress Scheduling Multicast Queues Egress Queues Unicast Crossbar Fabric Egress UPC Strict Priority + DWRR Scheduling Multicast 89

90 Multicast VOQ Unicast VOQ Increased Packet Buffer 25MB packet buffer is shared by every three 40GE ports or twelve 10GE ports. Nexus 5672: 150MB of total buffer Nexus 56128: 300MB of total buffer Nexus 6004: 800MB of total buffer Ingress UPC Egress UPC 16MB Unified Crossbar Fabric 9MB 90

91 Ingress Buffer - Flexible Buffer Management Shared buffer is good for burst absorption. Dedicated buffer is good for predicable performance for each port. N5600/6000 buffer management is flexible: dedicated plus shared. Long-distance FCoE, video editing (i.e., AVID), Big Data, and distributed storage UPC Ingress Buffer (16MB) Port 1 Dedicated SPAN Port 2 Dedicated Control Shared Packet Buffer ~14MB Port 3 Dedicated 91

92 Tune Shared vs Dedicated Buffer Allocation at Ingress queue-limit under network-qos policy specifies the dedicated buffer for each port and each class. Without queue-limit each class of service will get 100 KB of dedicated buffer. The size of dedicated buffer can be different for different classes of service. The policy applies to all ports in the chassis. Total ingress buffer minus the dedicated buffer and buffer for control and SPAN will be in the shared buffer pool. The following example sets the dedicated buffer for class-default to be 400 KB for all ports. switch(config)# policy-map type network-qos Policy-buffer switch(config-pmap-nq)# class type network-qos class-default switch(config-pmap-nq-c)# queue-limit bytes switch(config-pmap-nq-c)# system qos switch(config-sys-qos)# service-policy type network-qos Policy-buffer 92

93 Egress Buffer Design 9-MB buffer is reserved for egress traffic Unicast traffic can be buffered at egress and ingress. Multicast is buffered at egress in case of interface oversubscription. UPC Egress Buffer (9MB) Unicast Buffer Dedicated per port Multicast Buffer Shared ~6MB 93

94 Unicast Flow Control and Buffering Unicast is queued at egress buffer pool first and then ingress buffer after egress queue is full. Take advantage of ingress buffer from multiple port or ASIC for unicast burst absorption. Ensure fairness among multiple ingress ports 94

95 Identify Packet Drops Due to Congestion switch# show interface ethernet 1/7 Ethernet1/1 is down (SFP not inserted) Dedicated Interface <snip> RX <snip> 0 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble input discard 0 Rx pause TX 0 unicast packets 0 multicast packets 0 broadcast packets 0 output packets 0 bytes 0 jumbo packets 0 output error 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause Broadcast/Multicast/Unknown unicast flooding drop at egress Unicast drop at ingress switch# show queuing interface ethernet 1/7 Ethernet1/1 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 100 RX Queuing qos-group 0 Dedicated buffer for this class q-size: , HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: 0 Statistics: Pkts received over the port : Ucast pkts sent to the cross-bar : Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 0 Pkts discarded on ingress : Per-priority-pause status : Rx (Inactive), Tx (Inactive) Ingress drop for each class 95

96 Identify Congested Egress Port with VOQ Counters I X I A I X I A 1/7 1/10 Ingress UPC 3 Ingress UPC 4 Unified Crossbar Fabric Egress UPC 3 1/8 I X I A Using show interface CLI, we noticed drops on ingress ports How do we know which egress port is affected? 96

97 Step 1: Map Interface to UPC Number N6004# show hardware internal bigsur all-ports Bigsur Port Info: Port asic inst inst name idx slot asic eport logi flag adm opr if_index diag ucver sup b3 en dn pass 0.00 sup b3 en dn pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 en dn 1a pass gb1/ b3 dis dn 1a pass gb1/ b3 dis dn 1a pass gb1/ p 6 b3 en up 1a pass gb1/ p 7 b3 en up 1a pass gb1/ p 8 b3 en up 1a pass gb1/ p 9 b3 en up 1a pass gb1/ b3 dis dn 1a07a000 pass gb1/ p 11 b3 en up 1a07b000 pass 0.00 Done. 97

98 Step 2: Identify Congested Egress Port with VOQ Counters N6004# show platform software qd info counters voq asic-num port TRANSMIT TAIL DROP HEAD DROP Eth1/8 QUEUE Congested egress port and class of service Ingress UPC number N6004# show platform software qd info counters voq asic-num port TRANSMIT TAIL DROP HEAD DROP Eth1/8 QUEUE

99 Nexus 5600/6000 Architecture - Agenda Overview Performance Internal Architecture Transceivers/Cables Access Control List Analytics Multicast Quality of Service Applications 99

100 Nexus 5600/6000 Applications Compact Aggregation Nexus 7000 CORE Nexus 7000 Nexus 6004 L3 AGG. Nexus 6004/ Nexus 5600 vpc to Nexus 7000 or 6004 ACCESS Nexus 2000 FEX 100

101 Nexus 5600/6000 Applications Large Scale Fabric (Layer 2 or Layer 3) Nexus 7000 CORE L3 Nexus 6004 SPINE Nexus 6004/ Nexus 5600 FabricPath LEAF 101

102 Nexus 5600/6000 Applications High Performance Computing (HPC) Nexus 6004 CORE/AGG. L3/L2 Fabric Nexus 3548 Nexus 5600/6004 ACCESS 102

103 Nexus 5600/6000 Applications Multi hop FCoE FCoE FC Nexus 7000 CORE A B Nexus 7000 Nexus 6004 AGG Nexus 5600 Access FCoE FC 103

104 Nexus 5600/6000 Key Takeaways Rich Feature set Performance Scalability STP/vPC/FabricPath, FEX, L3, VXLAN, VM- FEX/Adapter-FEX Line rate L2 and L3 with low latency 256K MAC / 128K ARP / 32K LPM routes, 16K bridge domain with segment ID Innovations Intelligent SPAN, Buffer monitoring, Latency monitoring 104

105 Call to Action Visit the World of Solutions:- Cisco Campus Walk-in Labs Technical Solutions Clinics Meet the Engineer Lunch Time Table Topics, held in the main Catering Hall Recommended Reading: For reading material and further resources for this session, please visit 105

106 Complete Your Online Session Evaluation Complete your online session evaluation Complete four session evaluations and the overall conference evaluation to receive your Cisco Live T-shirt 106

107