IOS XR Platform Hardware Architectures

Transcription

1

2 BRKSPG-2404 IOS XR Platform Hardware Architectures LJ Wobker, Principal Engineer

3 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#brkspg Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Teaser Slide We re going to talk about how routers are actually built. We ll talk a lot about what things go into the design choices, and why it s really hard (actually, it s impossible) to build things that solve all the problems at the same time. What we re really going to talk about are engineering realities, and the tradeoffs that they impose on those of us who build, and those who ultimately use, the systems. We ll talk about the major dimensions that drive hardware design, and the relative costs to make one thing better in a given system. We will cover the high level architecture of several IOS XR platforms, but mostly in the context of how and why they are different and what design choices went in to each one Cisco and/or its affiliates. All rights reserved. Cisco Public

5 Agenda Introduction building a forwarding path Platform Design & Building Blocks IOS XR Platforms Hardware Virtualized Modern demands such as cloud computing, mobility, and media delivery are driving greater bandwidth and service requirements into provider networks. To address this, Cisco offers a broad portfolio of systems for provider networks. These systems all run the IOS XR operating system, but have meaningful differences in how they are designed from a system/hardware standpoint. This presentation discusses system architectures for the NCS-6000, ASR-9000, NCS-5500, and other IOS XR platforms with a specific focus on the high-level design decisions. A significant part of the talk is around silicon selection (custom vs. merchant, for example) and the mechanical and logical design of linecards for these larger systems.

6 Should I be here? Today s topics and not-topics. Yes! Hardware architectures System-level design Data planes Packet forwarding High-speed, complex, (expensive?) systems No! (or at least not much) Routing protocols / design Network level designs Control plane / OS infrastructure Selling a specific product BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 You can t always get what you want But if you try sometimes well you just might find you get what you need Nothing is free Some things are closer to free The further from the middle, the more things cost BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 The IOS XR Router Family CRS NCS 5000 NCS 5500 NCS 6000 ASR 9000 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Die size Liquid cooling Acoustics NEBS Heat Sinks Materials Signal Integrity Junction temp Cooling Airflow Chassis LLDRAM On-chip Serial DDR3 Backplane Filters Cable Management Fabric Connectors Routing Slot pitch SERDES Busbar Power PCB Interposer Process Silicon Fast Convergence Run to completion N:N QSFP28 Pipeline Optics PPS CPAK LR4 Features Programmability MPO Router Hardware Building Blocks IPoDWDM Stats Ops/sec Capacity Bandwidth FIB size Memory Buffering ACL scale TCAM GDDR5 HMC SRAM HBM BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 What s needed to build a forwarding path? 1. Optical to electrical 2. Transport a signal from optics to NPU 3. Ingress forwarding operations 4. Transport a signal from ingress NPU to egress NPU (fabric) 5. Egress forwarding operations 6. NPU to optics 7. Electrical to optical 2/6 3/5 4 1/7 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Logical view of forwarding path components LASERS / RECEIVERS FAST MEMORY SILICON Optics Tables SERDES TCAM MEMORY TCAM FAST MEMORY NPU TM Fabric Interface ASIC Optics Packet Buffers SILICON Fabric npower X1 Optics npower X1 FIA Optics npower X1 FIA Optics npower X1 FIA Optics npower X1 FIA SERDES DRAM CPU Ethernet Switch Control Ethernet BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Challenge: Scale routers faster than components Note: exponential scale 4096x 1024x 256x 64x 16x 4x 1x Moore s Law???? BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Technology Trends POS/Ethernet Standards Fastest Interface Bandwidth 4096x 1024x 256x 64x 16x Interface Speeds 10G 40G 100G Moore s Law 400G 4x 1x OC-48 GE OC BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Technology Trends POS/Ethernet Standards Fastest Interface & Router Bandwidth 4096x 1024x 256x 64x 16x 640G Interface Speeds 10G Buffered Router Bandwidth 40G Moore s Law 58T 24T 16T 8T 400G 2.4T 100G 4x 1x OC-48 GE 28G 8G OC BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Technology Trends - Optics Bandwidth/Volume 4096x Driven by physics, not Moore s Law, costs shifting SerDes dependency (NPU interface) Cooling challenge 30C lower max temp vs. ASICs 1024x 256x 64x 16x 4x 1x GBIC 10G XFP 10G SFP+ 100G CFP 100G CPAK 40G QSFP+ 100G QSFP BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Silicon & Modular Router PPS Per NPU/LC *Mpps for Standalone and Modular/Buffered 4096x 1024x 256x 64x 16x 4x 1x CPU 4 *represents a combination of memory, Silicon & SerDes x6 75 CRS x4 ASR CRS x4 280x5 NCS x4 ASR 9000 NCS 5500 ASR 9000 SoC no buffer Modular buffered BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Technology Trends Memory Commodity & Custom Bandwidth 4096x 1024x 256x 64x 16x 4x 1x 0.8 SDR 1.6 DDR Scaling FIB above ~256K IPv4 usec vs. msec buffering Operations / second critical for FIB DDR2 5 DDR3 10 ASR nd & 3 rd gen NCS GDDR5 HMC 28 NCS HBM DDR BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Technology Trends SerDes High-performance Electrical Link (Speed in GHz) 4096x 1024x 256x Optics to NPU NPU to fabric NPU to TCAM NPU to serial memory 64x 16x 4x 1x BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Technology Trends Compared 4096x 1024x 256x 64x Interfaces PPS 16x 4x Optics Memory SerDes 1x BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 Modular Buffered Router Bandwidth Systems growing much faster than components exponential scale 4096x 1024x 8T 24T 16T 58T 256x 2.4T 64x 640G Interfaces PPS 16x 4x 1x 8G 28G Optics Memory SerDes *similar trends for conductive materials, connectors, fans, power supplies, CPUs, BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Platform Design

22 Why not one platform? Requirements have a butterfly effect An individual requirement may dramatically impact architecture Every component ahead of the curve adds complexity Key Drivers Time / Cost Buffering FIB scale System Scale Features/Flexibility BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Time Terabit LCs in 2013 vs NCS T 200G NPU w/ full FIB, deep buffers, programmable Significant investment in npower X1 silicon development Custom memory for FIB and buffering extremely high bandwidth and ops/sec $271M acquisition of Lightwire for small, low-power 100G optics ASR T similar speed, flexibility, and power 3 years later Commercial NPU, customized for Cisco, commodity memories Cisco fabric, backward compatible with earlier generations QSFP28 optics, full range of chassis sizes NCS /2.4T lighter silicon architecture Commodity memory, partial bandwidth buffering, VoQ TCAM and non-tcam options (density change) Reduced counters, QoS, queueing complexity 3.6T with very high scale only in BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Buffering Ports On-chip limited to microseconds But Off-chip requires ASIC I/O pins fewer interfaces Board space and power Commodity bandwidth is limited Many devices may be needed Better suited for buffering than FIB (1 read/write per packet optimized for 128B read/write) Stalled High-end graphics and networking moving to custom Custom memories Fewer devices are needed to reach bandwidth, fewer pins Development and per-unit costs Extremely expensive Ports Ports NPU Ports Ports NPU FIB FIB Ports Ports Buffers BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 FIB Scale FIB On-chip FIB limited to ~256K LPM entries Usually an on-chip TCAM or lookup tree Can be scaled up to ~1M with well-known prefixes Combination of flat tables and LPM Ports NPU Buffers Ports Larger FIB tables require external memory: ASIC I/O pins redirected from interfaces to memory 4-8 reads for every packet high operations per second Board space (density) and power BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 System Scale Four common approaches FIB Fixed single NPU / Forwarding ASIC All resources dedicated to network interfaces Ports NPU Fabric May or may not have external memories Fixed multiple NPUs Buffers Connected via mesh or fabric chips Modular Expand with line cards Resources for fabric and usually external memories Multi-chassis Adds fiber connections to fabric cards (more power & board space) Increases software complexity BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 Network Interface Features and Flexibility Packet Processing Engine (PPE) C programmable Run to completion Anything is possible, no pure optimization Wide range of pipeline programmability NCS 5000 < NCS 5500 < ASR 9000 < NCS6K CRS PPES Packet rate is strongly correlated to cost, power, and flexibility Counting stuff is much harder than most people realize Ingress Port Link Term Parser Layer Tunnel VLAN Service Trans Term Fwd PMF Egress NCS 5500 PIPELINE FEC Res FEC Res BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Easy summary... Trying to build X things to solve X^N problems is the same thing as A bunch of engineering optimizations... and... A cut corner is the same thing as a tradeoff you didn t know you were making BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 IOS XR Platforms

30 The IOS XR Router Family CRS NCS 5000 NCS 5500 NCS 6000 ASR 9000 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 NCS 6000 First Dense 100G Highest router bandwidth capacity 128 Tbps first generation multi-chassis (2013) Only Tbps card on the market in With buffers and full FIB Dramatic improvement in power efficiency $6000/100G/month power savings over CRS-3 64x100 Merchant fabric Highly programmable forwarding, massive stats Custom memories for FIB and buffering Custom CPAK optics BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 NCS 6000 Line Card Architecture Slice architecture Optics, NPU & FIA per slice Run to Completion NPU w/ PPEs 1 Generation (so far) 1T line cards w/ 200G NPUs (40 nm) 400G NPUs in lab Optics Optics Optics T C A M Tables PPEs Packet Buffers npower X1 npower X1 T M Fabric Interface ASIC FIA TCAM for ACL/QoS scale Optics Optics npower X1 npower X1 FIA FIA Deep buffers (50+ msec) Optics npower X1 FIA DRAM CPU 10X 100GE BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 ASR 9000 Series Full range of chassis and interfaces 40G, 200G, and 1T generations Highly programmable forwarding Partnered for a customized NPU cisco software Highest scale, feature, and QoS capabilities Cisco fabric Commodity memories for FIB and buffering Mostly commodity optics BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 ASR 9000 System Design Backplane or midplane Flexible switch fabric options RP and fabric may be integrated into RSP Variable # of fabrics for increased capacity and redundancy First stage of fabric on line card Multiple airflow designs Front-to-back 9922, 9912, 9010 Side-to-back 9006 Side-to-side 9001, 9004 Modular options for lower-speed and legacy interfaces BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 ASR 9000 Line Card Architecture Slice architecture Optics, NPU & FIA per slice Flexible pipeline NPU 1 st stage of fabric on line card 3 Generations (so far) 40G line cards w/ 15G NPUs (90 nm) G line cards w/ 60G NPUs (55 nm) CPAK CPAK CPAK CPAK CPAK CPAK FIB TCAM NP-5c NP-5c NP-5c Packet Buffers FIA FIA FIA Fabric 800G-1.2T line cards with 240G NPUs (28 nm) (at least) one more generation of NPU/LC CPAK CPAK NP-5c FIA TCAM for ACL/QoS scale DRAM CPU Huge FIB (5M+) & buffers (200 msec) 8X 100GE BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 ASR 9000 Modular Line Card Architecture 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ PHY 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ PHY 2/4/8X 10GE 10G SFP+ 10G SFP+ PHY 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+ PHY 20X 10G Bay 0 FIB TCAM NP-5c FIB TCAM Packet Buffers FIA Packet Buffers Fabric 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP PHY 40G QSFP+ 40G QSFP+ PHY 40G QSFP+ PHY 100G CPAK PHY 100G CPAK 100G CPAK PHY Bay 1 NP-5c FIA DRAM CPU 1G SFP 20X 1G 1/2X 40G 1/2X100G MODULAR 3 RD GENERATION LC BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 ASR 9901 ~450Gbps, 2RU full featured system 2x NP5c NPUs, mini fabric Full feature set from ASR9k direct inheritance of all functionality / config Dense hierarchical QoS SFP(+) 10GE / 1GE x12 SFP 1GE x8 FIB NP-5c TCAM Packet Buffers FIA Packet Buffers Mixed set of interface speeds / optics MACSEC integrated into PHY full bandwidth deep packet buffers 2xQSFP28, 24xSFP+, 16xQSFP QSFP(+,28) 40/100GE QSFP(+,28) 40/100GE SFP(+) 10GE / 1GE x12 SFP 1GE x8 DRAM FIB NP-5c TCAM CPU Packet Buffers FIA Packet Buffers Fabric BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 NCS rd generation 100G Dense 100GE with deep buffers 8 & 16-slot modular, 1 & 2 RU fixed Up to 57.6 Tbps in ½ rack in T - 3.6T line cards Options for route and ACL scale via TCAM High scale 3.6T in 2018 Highly integrated Silicon Single ASIC for forwarding & fabric interface Dramatic power reduction to 0.24 W/Gbps QSFP28 optics GDDR5 commodity buffers BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 NCS 5500 System Design AIR INLET New chassis design for XR Orthogonal direct connect Horizontal line cards and vertical fabric Direct connection between line cards and fabric cards No midplane Distributed air intake between cards Fans cover fabric cards FAN REMOVED Provides cool air equally to all optics Optics require ~30C cooler operation than silicon Avoid preheating air from module to module REAR VIEW BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 NCS 5500 Line Card Architecture Slice architecture Optics & Pipelined Forwarding ASIC per slice integrated FIA Optional TCAM for FIB/ACL scale 1st Generation 3.6T line cards FA at 600G 2.9T line cards FA at 720G + TCAM Deep VoQ buffers (50+ msec) QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 Optics x 6 Optics x 6 Optics x 6 Optics x 6 Optics x 6 DRAM 36X 100GE Forwarding ASIC FA FA FA FA FA CPU Buffers BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ QSFP28 DRAM TCAM QSFP x 6 FA QSFP x 6 FA CPU Buffers QSFP28 QSFP28 QSFP28 DRAM TCAM CPU Buffers NCS 5501/5502 Architecture Forwarding ASIC NCS RU Single 800 Gbps Forwarding ASIC 4x 100G + 40x 10G NCS RU 8 600Gbps Forwarding ASICs Integrated switch fabric 48x 100G Optional TCAM for scale Deep VoQ buffers (50+ msec) 40x10G NCS G Switch Forwarding ASIC QSFP x 6 FA Fabric QSFP x 6 FA QSFP x 6 FA QSFP x 6 FA 4x100G NCS T Switch QSFP x 6 FA 48x100G 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

42 NCS55A1: 24 x 100GE 1 RU Fixed System: 24x QSFP28 ports Base version only (no room for TCAMs or MACSEC...) 2x 900 Gbps Forwarding ASICs (1200:900 nominal oversubscription) No Fabric ASIC, Forwarding ASICs are directly connected back-to-back BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 NCS 5000 Platforms Extending IOS XR for satellite, ToR, and beyond Maximizing Silicon capabilities No external memories small FIB, usec buffers All bandwidth to ports, no fabric Low cost and power Full IOS XR Routing NCS 5001 NCS 5002 Forwarding ASIC 32x100 FIB Buffers NCS 5011 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 CRS-X Scaling CRS into 100G 4x 100G 40x 100G 2x 100G + 5x 40G 3x 100G + 1x 100G IPoDWDM Up to 51.2T via multi-chassis Fully compatible with CRS-1 & CRS-3 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 Virtualized IOS XR ( XRv 9000 ) IOS XR on x86 hardware with Linux VM & containers Hosted & small PE with L2 & L3 VPNs Ideal route reflector 64 bit 10+ M routes Router creation in seconds rapid deployment Runs existing data plane forwarding code (recompiled) to an emulated NPU maintains consistent behavior / features BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Virtualized IOS XR Performance & QoS Targeted for 5-50 Gbps forwarding Power per bit looks awful except when it doesn t really matter! 20+ Gbps with features per socket 3-layer Hierarchical QoS Scale horizontally BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 Summary: Why not one platform? The requirement space is pretty broad Optimizing for everything optimizes for nothing But we have to optimize for some things or no one buys it! And we can t build an arbitrarily large suite of platforms So some of us are lucky enough to get paid to spend lots of time sitting around arguing about: which memory technologies, which silicon NPU architectures, which optical parameters, which mechanical designs, and a bunch of other things to trade off against each other... BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 One serious thing before we go I m grateful for the opportunity to be here and to talk about something (at least I think) is interesting. It s important to me personally that attendees find these presentations helpful and/or valuable. Certainly not everyone will enjoy every presentation -- but you guys spend a lot of money to come here and Cisco and the speakers do a lot of work to put on the event and the presentations. If the material is excellent, we ve done our jobs. If there s something we can do better, please let us know... I know you re not supposed to on the internet, but I really do read the comments. ;-) Thanks for your time today. Lawrence J Wobker, Cisco February 2018

49 Q & A

50 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#brkspg Cisco and/or its affiliates. All rights reserved. Cisco Public

51 Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public

52 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Tech Circle Meet the Engineer 1:1 meetings Related sessions BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Thank you

54

55