Cisco Nexus 9000 Architecture

Transcription

1

2 Cisco Nexus 9000 Architecture Tim Stevenson Distinguished Engineer, Technical Marketing CCIE 5561 Emeritus

3 Cisco Spark How Questions? Use Cisco Spark to chat with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space Cisco Spark spaces will be available until July 3, cs.co/ciscolivebot# 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 3

4 Session Abstract This session presents an in-depth study of the architecture of the latest generation of Nexus 9000 modular and top-of-rack data center switches. Topics include forwarding hardware, switching fabrics, and other physical design elements, as well as a discussion of key hardware-enabled features and capabilities that combine to provide high-performance data center network services Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 What This Session Covers Latest generation of Nexus 9000 switches with Cloud Scale ASICs Nexus 9500 modular switches with Cloud Scale linecards Nexus 9200 / Nexus 9300 Cloud Scale top-of-rack (TOR) switches System and hardware architecture, key forwarding functions, packet walks Not covered: First generation Nexus 9000 ASIC/platform architectures Nexus 9500 merchant-silicon based architectures Other Nexus platforms Catalyst 9000 platform 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Agenda Data Center and Silicon Strategy Cloud Scale Architecture Cloud Scale ASICs Forwarding and Features Cloud Scale Switching Platforms Packet Walks Key Takeaways 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Nexus 9000 Switching Portfolio Key Elements of the ASAP Data Center Nexus 9500 X9400 / X9400-S Merchant Broadcom XGS (Trident2+ / Tomahawk) Nexus 9500 X9600-R Merchant Broadcom DNX (Jericho) Nexus 9500 X9700-EX/FX Nexus 9200 / 9300-EX/FX Cisco Cloud Scale (ASE2 / ASE3 / LSE LS1800FX / S6400) Broadcom SOC solution Multi-chip architecture Wide industry availability Large forwarding tables Published SDK Deep packet buffer / VOQ Cell-based fabric Focus of this session Cisco SOC solution Rich forwarding feature set Smart buffers Advanced telemetry Optimized scale, cost, power 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Why Custom Silicon? Cisco competitive advantage vehicle for differentiating innovations Application Centric Infrastructure (ACI) policy model + congestion-aware flowlet switching Intelligent Buffers AFD / DPP Streaming telemetry: Flow Table for Tetration Analytics Flow table event notifications Streaming Statistics Export (SSX) Tight integration between hardware / software / marketing / sales / support Closely aligns hardware designs with software innovations, strategic product direction, competitive differentiators, serviceability 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 Agenda Data Center and Silicon Strategy Cloud Scale Architecture Cloud Scale ASICs Forwarding and Features Cloud Scale Switching Platforms Packet Walks Key Takeaways 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Cisco Cloud Scale ASIC Family Ultra-high port densities Reduces equipment footprint, enables device consolidation Multi-speed 1/10/25/40/50/100G Flexibility and future proofing Rich forwarding feature-set ACI, Segment Routing, singlepass VXLAN routing Flexible forwarding scale Single platform, multiple scaling alternatives Intelligent buffering Right-sized buffer with dynamic, advanced traffic optimization In-built analytics and telemetry Real-time network visibility for capacity planning, security, and debugging 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Cloud Scale 2016 ASE2 3.6T chip 6 slices of 6 x 100G each E-series fabric modules; 9200 series TORs ASE3 1.6T chip 2 slices of 8 x 100G each 92160YC-X TOR LSE 1.8T chip 2 slices of 9 x 100G each X9700-EX modular linecards; 9300-EX TORs Slice 0 600G Slice 3 600G Slice 1 600G Slice Interconnect Slice 4 600G Slice 2 600G Slice 5 600G ASE2 36 x 100G Slice 0 900G Slice Interconnect Slice 1 900G LSE 18 x 100G Slice 0 800G Slice Interconnect Slice 1 800G ASE3 16 x 100G 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Cloud Scale 2017 LS1800FX 1.8T chip 1 slice of 18 x 100G with MACSEC X9700-FX modular linecards; 9300-FX TORs S T chip 4 slices of 16 x 100G each 9364C TOR Slice 0 1.6T Slice 1 1.6T Slice 0 1.8T LS1800FX 18 x 100G Slice Interconnect Slice 2 1.6T Slice 3 1.6T S x 100G 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 What Is a Slice? Self-contained forwarding complex controlling subset of ports on single ASIC Separated into Ingress and Egress functions Ingress of each slice connected to egress of all other slices ASIC Slice Ingress Egress Slice Interconnect 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Slice Interconnect Provides non-blocking any-to-any interconnection between input and output of each slice Input Slice 1 Slice Interconnect Output Slice 1 Input Slice 2 Output Slice 2 Input Slice n Output Slice n 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Slice Forwarding Path Ingress Slice Ingress Forwarding Controller Ingress Packets Ingress MAC Packet Parser Packet Payload Lookup Key Lookup Pipeline Lookup Result Replication Slice Interconnect Egress Forwarding Controller Egress Packets Egress MAC Packet Rewrites Egress Policy Egress Buffering / Queuing / Scheduling Egress 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Ingress Lookup Pipeline From Ingress Packet Parser TCAM Flex Tiles TCAM To Egress Slice Lookup Key Forwarding Lookup Ingress Classification Load Balancing, AFD / DPP Lookup Result Flush Flow Table ASE3 / LSE / LS1800FX only Lookup Pipeline 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Flexible Forwarding Tiles Provide fungible pool of table entries for lookups Number of tiles and number of entries in each tile varies between ASICs Flex Tile Flex Tile Flex Tile Variety of functions, including: IPv4/IPv6 unicast longest-prefix match (LPM) IPv4/IPv6 unicast host-route table (HRT) IPv4/IPv6 multicast (*,G) and (S,G) MAC address table Adjacency table ACI policy Flex Tile Flex Tile Flex Tile Flex Tile Flex Tile Flex Tile 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Flexible Tiles by ASIC ASE2 20 x 16K + 4 x 8K tiles ASE3 20 x 16K + 4 x 8K tiles LSE 68 x 8K tiles S x 16K + 4 x 8K tiles LS1800FX 136 x 8K tiles 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Flex Tile Routing Templates Default Configurable forwarding templates determine flex tile functions system routing template syntax Templates as of NX-OS 7.0(3)I6(1): Default Dual-stack host scale* Internet peering* LPM heavy MPLS heavy* Multicast heavy Multicast NBM** Defined at system initialization reboot required to change profile LPM Heavy Multicast Heavy * Template does not support IP multicast Template not supported on modular Nexus 9500 ** Template not supported on TORs 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Agenda Data Center and Silicon Strategy Cloud Scale Architecture Cloud Scale ASICs Forwarding and Features Cloud Scale Switching Platforms Packet Walks Key Takeaways 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 IP Unicast Forwarding Router MAC match triggers L3 lookup Hardware performs exact-match on VRF and longest-match on IPDA Lookup result returns either adjacency pointer (index into MAC table), or ECMP pointer MAC table has output BD, rewrite MAC, and output port Receive (BD,DMAC) Router MAC Lookup (VRF,IPDA) Route Lookups ADJ_PTR What are the output BD, rewrite MAC, and output port? BD, MAC, port Transmit MAC Table Is the DMAC a Router MAC? HRT/LPM What s the longest match on the destination IP? ECMP_PTR ECMP ECMP Table MAC Table Which ECMP group should be used? 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 IP Tables Several methods for storing IP prefixes in hardware: HRT Hash table used for IPv4 /32 and IPv6 /128 host entries Provisioned from flex tiles LPM Combination of pivot and trie tiles, used for other prefix lengths Provisioned from flex tiles TCAM Handles overflow/hash collisions Traditional TCAM memory, front-ending flexible forwarding lookups 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Pivot / Trie Tiles for Scaling LPM Pivot tiles are hash tables containing base prefixes match base mask bits Trie tiles contain leaf entries for corresponding pivots match up to 3 least-significant prefix bits Combination of pivot hash lookup and corresponding trie bitmap lookup returns longest-match prefix entry Destination IP address from packet Base prefix mask shortest mask length in pivot tile s mask range /24 Pivot tile mask range: /24-/27 x.x.x.bbb (Unmasked bits) /24 (Masked bits) Hash result picks entry Hash Compare masked bits with Pivot entry contents Up to 3 unmasked bits Pivot Tile /24-/27 HIT! Prefix Trie Index Trie Tile Trie Offset Compare unmasked bits to trie entries HIT! Info Table ADJ_PTR / ECMP Index (from Pivot) + offset (from Trie) indexes Info Table to get Adjacency / ECMP pointer 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Trie Tile Lookup Trie tiles contain leaf entries for Pivot tile mask range: /24-/27 IP from packet Don t care Binary corresponding pivots Up to 15 prefixes can be packed into one trie entry Prefixes in FIB: IP Prefix Matched using hash lookup in pivot table (24 bits) Binary Matched using trie lookup (3 bits) / ******** / ******* / ****** / ***** * Trie lookup matches on these 3 bits 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 VXLAN Forwarding VXLAN and other tunnel encapsulation/ decapsulation performed in single pass Encapsulation L2/L3 lookup drives tunnel destination Rewrite block drives outer header fields (tunnel MACs/IPs/VNID, etc.) Decapsulation Packet parser determines whether and what type of tunnel packet Forwarding pipeline determines whether tunnel is terminated locally, drives inner lookups Receive Receive BD,DMAC VRF,IPDA Is this a tunnel packet? L2/L3 Lookups MAC/ LPM/HRT Parser Encapsulation Is packet destined to remote VTEP? DST_INTF/ ADJ_PTR (Outer VRF,IPDA) Tunnel Destination ADJ Decapsulation My TEP Table Is the tunnel destination a TEP I terminate? PTRs What are the tunnel header values? Outer MACs/ IPs/VNID Rewrite (Inner MAC/VRF/IP) Inner L2/L3 Lookups If Yes, process inner packet headers 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 Load Sharing Dynamic Load-Balancing (DLB) Supported on leaf switches in ACI fabric Congestion aware, flow-based or flowlet-based rebalances flows/flowlets based on path congestion Equal-Cost Multipath (ECMP) Static flow-based load-sharing Picks ECMP next-hop based on hash of packet fields and universal ID Source / destination IPv4 / IPv6 address (L3) Source / destination TCP / UDP ports (L4) L3 + L4 (default) GRE key field 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 DLB versus ECMP Load-Sharing Add port DLB Flow Hash from Packet Flowlet Table Add flowlet New flowlet DLB Candidates Dynamic Rate Estimator Least congested link(s) Output Port Transmit 64K entries Existing flowlet Output Port ECMP Pointer from Lookup ECMP Flow Hash from Packet ECMP Link Count from IP Lookup Hash % Link Count Modulo operation Base Offset ECMP Links ECMP Table Output Port Transmit 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Multicast Forwarding Multicast source and group forwarding entries populated in HRT Additional, secondary table for multicast also provisioned ( MC_INFO ) from flex tiles MET table in egress slice holds output interface list (OIL) Receive (VRF,IPSA) (VRF,IPDA) Route Lookups PTR RPF MET MET_Index Replication List Transmit HRT MC_INFO MET Is there a (*,G) or (S,G) match? Did this packet pass RPF check? Where s the OIL for this mroute stored? Which interfaces are part of the OIL and require replication? 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 Classification TCAM Dedicated TCAM for packet classification Capacity varies depending on platform Leveraged by variety of features: RACL / VACL / PACL L2/L3 QOS SPAN / SPAN ACL NAT COPP Flow table filter (LS1800FX) Ingress Slice Egress Slice ASE2 / ASE3 / LSE 4K ingress ACEs / 2K egress ACEs Ingress Slice Egress Slice LS1800FX / S6400 5K ingress ACEs / 2K egress ACEs 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 TCAM Region Resizing Default carving allocates 100% of TCAM and enables: Ingress / Egress RACL Ingress QOS SPAN SPAN ACLs Flow table filter (LS1800FX only) Reserved regions Based on features required, user can resize TCAM regions to adjust scale To increase size of a region, some other region must be sized smaller Region sizes defined at initialization changing allocation requires system reboot Configure all regions to desired size ( hardware access-list tcam region ), save configuration, and reload RACL RACL RACL RACL RACL RACL RACL RACL RACL QOS QOS QOS SPAN SPAN SACL SACL FT FT RSVD RSVD Ingress Slice Egress Slice RACL RACL RACL RACL RACL RACL RACL RSVD 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 Flow Table ASE3 / LSE / LS1800FX platforms support hardware flow table 32K flow table entries per slice Collects full flow information plus metadata for: Tetration Analytics Triggered Flow Table Events (FTE) Netflow v9 Direct hardware export for Tetration / FTE 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 Flow Table and Flow Table Events Flow table operation: 1. Determine if FT/FTE enabled for flow 2. Install FT record; capture FTE records if triggered 3. Flush FT / FTE records, encapsulate in IP/UDP 4. Submit packet for lookup Submit for forwarding lookup Is export interval reached? Flush to collector(s) Admitted flows Flow Table Flow Key Flow Records Filter TCAM Event notification Is event FIFO full (or timer popped)? Did this flow trigger one or more events? Is flow collection and/or flow events enabled for this flow? FTE Records FTE FIFO Flush to collector(s) Ingress Flows Parser Lookup Key Lookup Pipeline Lookup Result 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Flow Table Events Event triggers: Packet value match Buffer/ACL drop Forwarding exception Latency threshold Microburst threshold Analytics changed Got it! I see queue drops but who s affected?! Flow Table Event Record Flow Data Timestamp Port Queue Buffer Drop Packet drops! Enable Flow Table Event on packet buffer drops 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 Streaming Statistics Export (SSX) Streams statistics and other ASIClevel data Direct export from ASIC no switch CPU involvement User defines streaming parameters which statistics, how often, and to which collector Hardware support only in Nexus 9364C switch (S6400 based) S6400 Submit for forwarding lookup SSX Parser Periodically pull values from ASIC Stats Tables Registers Lookup Pipeline SSX Packet L2/L3 Headers Metadata TLV-1 TLV-2 TLV-n CRC 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 Buffering Cloud Scale platforms implement shared-memory egress buffered architecture Each ASIC slice has dedicated buffer only ports on that slice can use that buffer Dynamic buffer protection adjusts max thresholds based on class and queue occupancy Intelligent buffer options maximize buffer efficiency Slice 0 5.1MB Slice 3 5.1MB Slice 1 5.1MB Slice Interconnect Slice 4 5.1MB Slice 2 5.1MB Slice 5 5.1MB Slice MB Slice Interconnect Slice MB Slice MB Slice Interconnect Slice MB Slice MB Slice Interconnect Slice MB Slice MB Slice MB Slice MB ASE2 5.1MB/slice (30.6MB total) ASE3 10.2MB/slice (20.4MB total) LSE 18.7MB/slice (37.4MB total) S MB/slice (40.8MB total) LS1800FX 40.8MB/slice (40.8MB total) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 Intelligent Buffering Innovative Buffer Management for Cloud Scale switches Dynamic Buffer Protection (DBP) Controls buffer allocation for congested queues in shared-memory architecture Approximate Fair Drop (AFD) Maintains buffer headroom per queue to maximize burst absorption Dynamic Packet Prioritization (DPP) Prioritizes short-lived flows to expedite flow setup and completion Miercom Report: Speeding Applications in Data Center Networks Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 Dynamic Buffer Protection (DBP) Prevents any output queue from consuming more than its fair share of buffer in shared-memory architecture Defines dynamic max threshold for each queue If queue length less than threshold, packet is admitted Otherwise packet is discarded Threshold calculated by multiplying free memory by configurable Alpha (α) value (weight) queue-limit dynamic alpha-value in queuing policy 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 Buffer in MB Buffer in MB Buffer in MB Alpha Parameter Examples Default Alpha on Cloud Scale switches Alpha (α) = 0.5 Alpha (α) = 1 Alpha (α) = Buffer per queue == Buffer per queue == 30 free buffer 30 ½ free buffer Buffer per queue == 14 x free buffer Number of Oversubscribed Queues Number of Oversubscribed Queues Number of Oversubscribed Queues Buffer per queue (MB) Free buffer (MB) Buffer per queue (MB) Free buffer (MB) Buffer per queue (MB) Free buffer (MB) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 Buffering Ideal versus Reality Ideal buffer state Actual buffer state Buffer available for burst absorption Buffer available for burst absorption Buffer consumed by sustained-bandwidth TCP flows Buffer consumed by sustained-bandwidth TCP flows Sustained-bandwidth TCP flows back off before all buffer consumed Sustained-bandwidth TCP flows consume all available buffer before backing off 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 Approximate Fair Drop (AFD) Maintain throughput while minimizing buffer consumption by elephant flows keep buffer state as close to the ideal as possible 1. Distinguish elephant flows from other flows 2. Track elephant flows and adjust AFD drop probability 3. Enforce AFD at egress queue AFD desired queue depth Receive Flow data from packet Hash Index into Elephant Trap 8K flows Elephant Trap Track Elephants 1K flows Elephants Table Drop probability Egress Queuing Transmit Does this flow exceed Elephant threshold? Monitor Elephant flow bandwidth, set AFD drop probability Has this queue reached AFD queuedesired watermark? Drop 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 Dynamic Packet Prioritization (DPP) Prioritize initial packets of new / short-lived flows Up to first 1K packets assigned to higher-priority qos-group Receive Flow data from packet Hash Index into Prioritization Flow Table 64K flows Prioritization Flow Table No Drive new priority Egress Queuing Transmit Has this flow exceeded prioritization threshold? Yes Maintain original priority 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 Queuing and Scheduling UC0 MC0 UC1 MC1 UC2 MC2 UC3 MC3 UC4 MC4 UC5 MC5 UC6 MC6 UC7 MC7 50/50 DWRR CPU Class 0 Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 SPAN Strict Priority Configurable Weights / Priority Egress Port Best Effort Final Winner 8 user classes and 16 queues per output port (8 unicast, 8 multicast) QOS-group drives class; egress queuing policy defines class priority and weights Dedicated classes for CPU traffic and SPAN traffic 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 Ingress QOS / Egress Queuing Policies Default QOS behavior: All user data goes to q-default Trust received QOS markings To select egress queue, use set qos-group in ingress QOS policy To set/change packet markings, use set cos / precedence / dscp in ingress QOS policy To change queuing behavior, manipulate egress queuing policies Ingress QOS policy Egress Queuing policy Ingress set qosgroup set dscp Selects queue queue-limit priority bandwidth AFD shape Egress set prec Marks packet set cos 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 Agenda Data Center and Silicon Strategy Cloud Scale Architecture Cloud Scale ASICs Forwarding and Features Cloud Scale Switching Platforms Packet Walks Key Takeaways 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Cloud Scale Platforms Nexus 9200 Entry-level TOR platform Based on ASE2/ASE3 ASICs Core Cloud Scale functionality Some limitations: No ACI support No Flow Table No FEX support Nexus 9300, 9300-EX, and 9300-FX Premier TOR platform Based on LSE / LS1800FX / S6400 ASICs Full Cloud Scale functionality New FX option with MACSEC using LS1800FX silicon Nexus 9500 X9700-EX and X9700-FX Modules Switching modules for Nexus 9500 chassis Based on LSE / LS1800FX ASICs Full Cloud Scale functionality ACI spine / standalone aggregation or spine FX option with MACSEC using LS1800FX silicon 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 Nexus 9200 Cloud Scale TOR Switches 36-port 100G QSFP28 N9K-C9236C ASE2-based NX-OS: 7.0(3)I4(1) 72-port 40G QSFP N9K-C9272Q ASE2-based NX-OS: 7.0(3)I3(1) 56-port 40G QSFP + 8-port 100G QSFP28 N9K-C92304QC ASE2-based NX-OS: 7.0(3)I4(1) Key Features NX-OS mode only Cloud Scale L2/L3 feature set High-density fixed 40G/100G aggregation High-performance 10G/25G server access 48-port 10/25G + 18-port 100G QSFP28 N9K-C92300YC ASE2-based NX-OS: 7.0(3)I6(1) 48-port 10/25G SFP port 40G QSFP + 2-port 100G QSFP28 N9K-C92160YC-X ASE3-based NX-OS: 7.0(3)I3(1) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 Nexus 9200 Switch Architectures CPU ASE2 CPU ASE Front Panel Ports 1-72 Front Panel Ports C9236C (100G) C9272Q (40G) CPU ASE Front Panel Ports C92160YC-X (10/25G + 40G + 100G) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 Nexus 9300-EX Cloud Scale TOR Switches 48-port 10/25G SFP port 100G QSFP28 N9K-C93180YC-EX LSE-based ACI: 1.3(1) NX-OS: 7.0(3)I4(2) 48-port 1/10GBASE-T + 6-port 100G QSFP28 N9K-C93108TC-EX LSE-based ACI: 2.0(1) NX-OS: 7.0(3)I4(2) Key Features Dual capability ACI and NX-OS mode Flexible port configurations 1/10/25/40/50/100G Native 25G server access ports Flow Table for Netflow and Tetration Analytics Smart buffer capability (AFD / DPP) 32-port 40G/50G/100G QSFP28 N9K-C93180LC-EX LSE-based ACI: 2.2(1) NX-OS: 7.0(3)I6(1) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 Nexus 9300-EX Switch Architectures CPU LSE 1-48 Front Panel Ports C93180YC-EX (10/25G + 100G) / C93108TC-EX (10G + 100G) CPU LSE 1-28 Front Panel Ports C93180LC-EX (40/50G + 100G) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 Nexus 93180LC-EX Port Speed Configuration ACI x 40G + 6 x 100G ACI default (40G leaf) 6x100G (25,27,29-32) always function as spine uplinks x 100G 100G leaf option 6x100G (25,27,29-32) always function as spine uplinks 40G 100G Disabled 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 Nexus 93180LC-EX Port Speed Configuration Standalone x 40G + 4 x 100G Standalone default Ports 29-32: 100G, 40G, or 4x10G breakout Ports 1-28: 10G (QSA)/40G/50G Ports 1,3,5..27: 4x10G breakout (disables lower port) x 40G + 6 x 100G Ports 25,27,29-32: 100G, 40G, or 4x10G breakout Ports 1-24: 10G (QSA)/40G/50G Ports 1,3,5..23: 4x10G breakout (disables lower port) Ports 1,3,5,..27,29-32: 100G, 40G, or 4x10G breakout 18 x 100G 40G 100G Disabled 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Nexus 9300-FX Cloud Scale TOR Switches Pervasive MACSEC Key Features 48-port 10/25G SFP port 100G QSFP28 N9K-C93180YC-FX LS1800FX-based ACI: 2.2(2e) NX-OS: Roadmap 48-port 1/10GBASE-T + 6-port 100G QSFP28 N9K-C93108TC-FX LS1800FX-based ACI: 2.2(2e) NX-OS: Roadmap Dual capability ACI and NX-OS mode Flexible port configurations 1/10/25/40/50/100G Line-rate -bit encryption all ports 32G FC support on all SFP ports 25G distances beyond 3m (RS-FEC) Flow Table for Netflow and Tetration Analytics Smart buffer capability (AFD / DPP) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 Nexus 9300-FX Switch Architectures CPU LS1800FX 1-48 Front Panel Ports C93180YC-FX (10/25G + 100G) / C93108TC-FX (10G + 100G) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 Nexus 9364C 100G Cloud Scale Switch Key Features 64-port 100G QSFP28 N9K-C9364C S6400-based ACI/NX-OS: Roadmap Dual capability ACI and NX-OS mode Compact, high-performance fixed ACI spine 100G/50G/40G/10G (single port mode no breakout) MACSEC/CloudSec on 16 ports Streaming Statistics Export (SSX) Smart buffer capability (AFD / DPP) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 Nexus 9364C Switch Architecture CPU S6400 CloudSec 1 CloudSec 2 CloudSec 3 CloudSec 4 CloudSec 5 CloudSec 6 CloudSec 7 CloudSec Front Panel Ports C9364C (100G) Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 Nexus 9500 Modular Cloud Scale Switches + + Nexus 9504 Nexus 9508 Nexus 9516 Common Equipment E-Series Fabric Modules EX / FX Series Line Cards 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 X9700-EX 100G Cloud Scale Modules N9K-X9732C-EX / N9K-X9736C-EX Line-rate performance up to 3.2Tbps capacity Advanced features Tetration Analytics hardware sensor Smart buffer capability (AFD / DPP) Flexible forwarding tables VXLAN routing 32-port 100G QSFP28 X9732C-EX LSE-based ACI: 1.3(1) NX-OS: 7.0(3)I4(2) 32 / 36 x QSFP28-based 100G ports Pin-compatible with 40G QSFP+ Flexible speed ports 1 / 10 / 25 / 40 / 50 / 100G capability Supported in ACI and NX-OS standalone mode 36-port 100G QSFP28 X9736C-EX LSE-based ACI: Roadmap NX-OS: 7.0(3)I6(1) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 N9K-X9732C-EX / N9K-X9736C-EX Architecture 8 x 100G with speedup EOBC To Fabric Modules EOBC To Fabric Modules LC CPU LSE 1 LSE 2 LSE 3 LSE 4 LC CPU LSE 1 LSE 2 LSE 3 LSE Front Panel Ports Front Panel Ports 8 x 100G front-panel ports per LSE X9732C-EX 9 x 100G front-panel ports per LSE X9736C-EX 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 X9700-EX 10/25G + 100G Cloud Scale Module N9K-X97160YC-EX 1.6Tbps capacity with line-rate performance Advanced features Tetration Analytics hardware sensor Smart buffer capability (AFD / DPP) Flexible forwarding tables VXLAN routing 48p 10/25G SFP+ and 4p 100G QSFP28 X97160YC-EX LSE-based NX-OS: 7.0(3)I5(2) 48 x SFP28-based 25G ports Pin-compatible with 1G SFP and 10G SFP+ Flexible speed ports 1 / 10 / 25G capability 4 x QSFP28-based 100G ports Pin-compatible with 40G QSFP+ Flexible speed ports 1 / 10 / 25 / 40 / 50 / 100G capability Supported in NX-OS standalone mode only 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 N9K-X97160YC-EX Architecture 8 x 100G with speedup EOBC To Fabric Modules LC CPU LSE 1 LSE Front Panel Ports X97160YC-EX 24 x 10/25G and 2 x 100G front-panel ports per LSE 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 X9700-FX 100G Cloud Scale Module N9K-X9736C-FX 3.2Tbps capacity line-rate performance at 170-byte frames Advanced features Line-rate MACSEC on all ports CloudSec encryption (8 ports) Tetration Analytics hardware sensor Smart buffer capability (AFD / DPP) Flexible forwarding tables VXLAN routing 36p 100G QSFP28 X9736-FX LS1800FX-based ACI: Roadmap NX-OS: Roadmap 36 x QSFP28-based 100G ports Pin-compatible with 40G QSFP+ Flexible speed ports 1 / 10 / 25 / 40 / 50 / 100G capability Supported in ACI and NX-OS standalone mode 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 N9K-X9736C-FX Architecture EOBC To Fabric Modules 8 x 100G with speedup LC CPU LS1800FX 1 LS1800FX 2 LS1800FX 3 LS1800FX 4 CloudSec 1 CloudSec 2 CloudSec 3 CloudSec x 100G front-panel ports per LS1800FX Front Panel Ports X9736C-FX 7 x 100G MACSEC capable ports per LS1800FX 2 x 100G MACSEC / CloudSec capable ports per LS1800FX 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 MACSEC Hardware Encryption Provides link-level hop-by-hop encryption IEEE 802.1AE 128-bit and -bit AES encryption with MKA Key Exchange Native hardware support available on: All ports on X9736C-FX linecard All ports on Nexus 93180YC-FX / 93108TC-FX switches 16 x 100G ports on Nexus 9364C switch MACSEC Frame Format DMAC SMAC DMAC SEC-Tag SMAC EType EType Payload Payload ICV FCS FCS Original Frame Encrypted Frame 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 64

65 CloudSec Hardware Encryption Provides VTEP-to-VTEP encryption Encrypts VXLAN header and payload for transport over arbitrary IP network Hardware support available on: 8 x 100G ports on X9736C-FX linecard 16 x 100G ports on Nexus 9364C No support on other TOR switches CloudSec Frame Format DMAC SMAC DMAC EType SMAC IP EType UDP* IP SEC-Tag UDP VXLAN VXLAN Payload Payload ICV FCS FCS Original VXLAN Packet Encrypted Packet * CloudSec UDP dest port 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 Cloud Scale Fabric Modules 9504-FM-E 9508-FM-E 9516-FM-E ASE2 ASE2 1 ASE2 2 ASE2 1 ASE2 2 ASE2 3 ASE x 100G 32 x 100G 32 x 100G 64 x 50G 64 x 50G 64 x 50G 64 x 50G Cloud Scale linecards require Cloud Scale fabric modules Provide up to 3.2Tbps capacity per IO module slot with 4 FMs Note: Cloud Scale FMs support X9700-EX and X9700-FX modules only N9K-C9504-FM-E ACI: 1.3(1) NX-OS: 7.0(3)I4(2) N9K-C9508-FM-E ACI: 1.3(1) NX-OS: 7.0(3)I4(2) N9K-C9516-FM-E ACI: Roadmap NX-OS: 7.0(3)I5(2) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 Cloud Scale Fabric Connectivity Nexus 9504 FM2 FM3 4 x 9504-FM-E FM4 FM6 32 x 100G per FM / 4 slots = 800G per slot per FM ASE2 ASE2 ASE2 ASE2 8 x 100G for each LSE/LS1800FX 800G x 4 FM x 4 slots = 12.8T per system 2 x 100G to each ASE2 32 x 100G per module = 3.2T / slot LSE 1 LSE 2 LSE 3 LSE 4 LSE 1 LSE 2 LS1800FX 1 LS1800FX 2 LS1800FX 3 LS1800FX 4 X9732C-EX / X9736C-EX 32 x 100G per module = 3.2T / slot X97160YC-EX 16 x 100G per module = 1.6T / slot X9736C-FX 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Cloud Scale Fabric Connectivity Nexus x 9508-FM-E ASE2 FM2 ASE2 ASE2 FM3 ASE2 ASE2 FM4 ASE2 ASE2 FM6 ASE2 64 x 100G per FM / 8 slots = 800G per slot per FM 8 x 100G for each LSE/LS1800FX 800G x 4 FM x 8 slots = 25.6T per system 1 x 100G to each ASE2 32 x 100G per module = 3.2T / slot LSE 1 LSE 2 LSE 3 LSE 4 LSE 1 LSE 2 LS1800FX 1 LS1800FX 2 LS1800FX 3 LS1800FX 4 X9732C-EX / X9736C-EX 32 x 100G per module = 3.2T / slot X97160YC-EX 16 x 100G per module = 1.6T / slot X9736C-FX 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 Cloud Scale Fabric Connectivity Nexus 9516 FM2 FM3 4 x 9516-FM-E FM4 FM6 x 50G per FM / 16 slots = 800G per slot per FM ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 ASE2 1 x 50G to each ASE2 800G x 4 FM x 16 slots = 51.2T per system 64 x 50G per module = 3.2T / slot LSE 1 LSE 2 LSE 3 LSE 4 LSE 1 LSE 2 LS1800FX 1 LS1800FX 2 LS1800FX 3 LS1800FX 4 X9732C-EX / X9736C-EX 16 x 50G for each LSE/LS1800FX 64 x 50G per module = 3.2T / slot X97160YC-EX 32 x 50G per module = 1.6T / slot X9736C-FX Note: 50G flow limit in Nexus 9516 chassis 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 Agenda Data Center and Silicon Strategy Cloud Scale Architecture Cloud Scale ASICs Forwarding and Features Cloud Scale Switching Platforms Packet Walks Key Takeaways 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 Packet Walk (TOR) IP Unicast 93180YC-EX Receive from Slice 0, transmit to Slice 1 Rewrite SMAC/DMAC TTL decrement Write QOS fields LSE Slice Interconnect MAC IFC Packet Parser Lookup Pipeline Buffering / Queuing / Scheduling EFC Egress Policy Rewrites MAC Slice 0 Slice 1 Receive frame FCS checking VLAN checking Extract header fields Generate lookup key MTU checks Longest-match prefix lookup Adjacency/ECMP pointer DLB/ECMP decision Ingress policy enforcement AFD/DPP Flow table / FTE Receive from slice interconnect Buffer packet in queue (DBP) AFD drops Scheduling Egress policy enforcement FCS generation CRC stomp Transmit frame 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 Packet Walk (TOR) Multicast MET lookup Replication to local OIL 9364C S6400 Slice 2 Replication Slice 3 Slice with no receivers drops packet Buffering / Queuing / Scheduling Buffering / Queuing / Scheduling EFC Egress Policy Rewrites MAC Replication to all slices Slice Interconnect (*,G) or (S,G) lookup RPF check MET pointer MAC IFC Packet Parser Lookup Pipeline Buffering / Queuing / Scheduling EFC Egress Policy Rewrites MAC Slice 0 Replication Slice Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 Packet Walk (Modular) Multicast (*,G) or (S,G) lookup MET pointer Replication to all slices FM ASE2 Slice Interconnect Multicast replication to local OIFs (fabric links) MET lookup Replication to local OIL (front panel and/or fabric) Slice 0 Slice 1 Slice 2 Slice 3 Slice 4 Slice 5 (*,G) or (S,G) lookup MET pointer LSE 1 LSE 2 LSE 3 LSE 4 LSE 1 LSE 2 Slice 1 Slice 1 Slice 1 Slice Interconnect Slice Interconnect Replication to all slices Slice Interconnect Slice 0 Slice 0 Slice 0 Replication to all slices (*,G) or (S,G) lookup RPF check MET pointer X9732C-EX / X9736C-EX Multicast replication to local OIFs X97160YC-EX 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 Packet Walk VXLAN Encapsulation Add L2 / IP / UDP / VXLAN header 93180YC-EX LSE Slice Interconnect MAC IFC Packet Parser Lookup Pipeline Buffering / Queuing / Scheduling EFC Egress Policy Rewrites MAC Slice 0 Slice 1 L2/L3 lookup Adjacency pointer Remote tunnel endpoint ECMP decision 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 Packet Walk VXLAN Decapsulation 93180YC-EX LSE Slice Interconnect MAC EFC Rewrites Egress Policy Buffering / Queuing / Scheduling IFC Lookup Pipeline Packet Parser MAC Slice 0 Slice 1 Remove outer headers Rewrite SMAC/DMAC Decrement TTL Destination TEP lookup Inner L2/L3 lookups Adjacency/ECMP pointer ECMP decision Tunnel check Extract header fields Generate lookup key 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 Agenda Data Center and Silicon Strategy Cloud Scale Architecture Cloud Scale ASICs Forwarding and Features Cloud Scale Switching Platforms Packet Walks Key Takeaways 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Nexus 9000 Momentum in the Marketplace 12,000+ 3, $3B Nexus 9K Customers Globally ACI Customers Ecosystem Partners Business Run Rate ECOSYSTEM PARTNERS 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 Key Takeaways You should now have a thorough understanding of the Nexus 9000 Cloud Scale switching platform architecture Feature-rich, innovative switching platform addresses virtually every deployment scenario Nexus 9000 Cloud Scale platform forms foundation of the ASAP Data Center 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card. Complete your session surveys through the Cisco Live mobile app or on Don t forget: Cisco Live sessions will be available for viewing on demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 Related Sessions and Resources BRKDCN-3101 Nexus 9000 (Standalone) Architecture Brief and Troubleshooting BRKDCN-3346 End-to-End QoS Implementation and Operation with Cisco Nexus Switches BRKDCN-3020 Network Analytics using Nexus 3000/9000 Switches BRKACI-2040 Tetration Analytics BRKDCN-3378 Building Data Center Networks with VXLAN BGP-EVPN LTRDCN-2223 Implementing VXLAN in a Data Center (Lab) 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 Thank you 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83