IOS XR Platform Hardware Architectures

Transcription

1

2 IOS XR Platform Hardware Architectures LJ Wobker, Principal Engineer Lane Wigley, Technical Marketing BRKSPG-2404

3 Agenda/Abstract Introduction building a forwarding path Platform Design & Building Blocks IOS XR Platforms Hardware Virtualized Modern demands such as cloud computing, mobility, and media delivery are driving greater bandwidth and service requirements into provider networks. To address this, Cisco offers a broad portfolio of systems for provider networks. These systems all run the IOS XR operating system, but have meaningful differences in how they are designed from a system/hardware standpoint. This presentation discusses system architectures for the NCS-6000, ASR-9000, NCS-5500, and other IOS XR platforms with a specific focus on the high-level design decisions. A significant part of the talk is around silicon selection (custom vs. merchant, for example) and the mechanical and logical design of linecards for these larger systems.

4 Should I be here? Today s topics and not-topics. Yes! Hardware architectures System-level design Data planes Packet forwarding High-speed, complex, (expensive!) systems No! (or at least not much) Routing protocols / design Network level designs Control plane / OS infrastructure Selling a specific product BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 You can t always get what you want But if you try sometimes well you just might find you get what you need Nothing is free Some things are closer to free The further from the middle, the more things cost BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 The IOS XR Router Family CRS NCS 5000 NCS 5500 NCS 6000 ASR 9000 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 What s needed to build a forwarding path? 1. Optical to electrical 2. Transport a signal from optics to NPU 3. Ingress forwarding operations 4. Transport a signal from ingress NPU to egress NPU (fabric) 5. Egress forwarding operations 6. NPU to optics 7. Electrical to optical 2/6 3/5 4 1/7 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Logical view of forwarding path components LASERS / RECEIVERS FAST MEMORY SILICON Optics Tables SERDES TCAM MEMORY TCAM FAST MEMORY NPU TM Fabric Interface ASIC Optics Packet Buffers npower X1 SILICON Fabric Optics npower X1 FIA Optics npower X1 FIA Optics npower X1 FIA Optics npower X1 FIA SERDES DRAM CPU Ethernet Switch Control Ethernet BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Die size Liquid cooling Acoustics NEBS Heat Sinks Materials Signal Integrity Junction temp Cooling LLDRAM On-chip Serial DDR3 Airflow Chassis Backplane Filters Cable Management Fabric Connectors Routing Slot pitch SERDES Busbar Power PCB Interposer Process Silicon Fast Convergence Run to completion N:N QSFP28 Pipeline Optics PPS CPAK LR4 Features Programmability MPO Router Hardware Building Blocks Stats IPoDWDM Ops/sec Capacity Bandwidth FIB size HMC SRAM Buffering TCAM ACL scale GDDR5 Memory HBM BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 9

10 Challenge: Scale routers faster than components Note: exponential scale 4096x 1024x 256x 64x 16x 4x 1x Moore s Law???? BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Technology Trends POS/Ethernet Standards Fastest Interface Bandwidth 4096x 1024x 256x 64x 16x Interface Speeds 10G 40G 100G Moore s Law 400G 4x 1x OC-48 GE OC BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Technology Trends POS/Ethernet Standards Fastest Interface & Router Bandwidth 4096x 1024x 256x 64x 16x 640G Interface Speeds 10G Buffered Router Bandwidth 40G 8T 2.4T 100G Moore s Law 24T 16T 58T 400G 4x 1x OC-48 GE 28G 8G OC BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Technology Trends - Optics Bandwidth/Volume 4096x 1024x 256x 64x 16x Driven by physics, not Moore s Law, costs shifting SerDes dependency (NPU interface) Cooling challenge 30C lower max temp vs. ASICs 4x 1x GBIC 10G XFP 100G CFP 100G CPAK 40G QSFP+ 100G QSFP BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Silicon & Modular Router PPS Per NPU/LC *Mpps for Standalone and Modular/Buffered 4096x *represents a combination of memory, Silicon & SerDes 1024x x 64x 16x 4x 1x CPU CRS-1 14x4 ASR 9000 CRS x4 ASR x6 280x5 NCS 5500 NCS x4 ASR 9000 SoC no buffer Modular buffered BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Technology Trends Memory Commodity & Custom Bandwidth 4096x 1024x 256x 64x 16x 4x 1x 0.8 SDR 1.6 DDR Scaling FIB above ~256K IPv4 usec vs. msec buffering Operations / second critical for FIB DDR2 5 DDR3 10 ASR nd & 3 rd gen NCS GDDR5 HMC 28 NCS HBM DDR BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Technology Trends SerDes High-performance Electrical Link (Speed in GHz) 4096x 1024x 256x Optics to NPU NPU to fabric NPU to TCAM NPU to serial memory 64x 16x 4x 1x BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Technology Trends Compared 4096x 1024x 256x 64x Interfaces PPS 16x 4x Optics Memory SerDes 1x BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Modular Buffered Router Bandwidth Systems growing much faster than components exponential scale 4096x 1024x 8T 24T 16T 58T 256x 2.4T 64x 640G Interfaces PPS 16x 4x 1x 8G 28G Optics Memory SerDes *similar trends for conductive materials, connectors, fans, power supplies, CPUs, BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 Platform Design

20 Why not one platform? Requirements have a butterfly effect An individual requirement may dramatically impact architecture Every component ahead of the curve adds complexity Key Drivers Time / Cost Buffering FIB scale System Scale Features/Flexibility BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 Time Terabit LCs in 2013 vs NCS T 200G NPU w/ full FIB, deep buffers, programmable Significant investment in npower X1 silicon development Custom memory for FIB and buffering extremely high bandwidth and ops/sec $271M acquisition of Lightwire for small, low-power 100G optics ASR T similar speed, flexibility, and power 3 years later Commercial NPU, customized for Cisco, commodity memories Cisco fabric, backward compatible with earlier generations QSFP28 optics, full range of chassis sizes NCS /2.4T lighter silicon architecture Commodity memory, partial bandwidth buffering, VoQ TCAM and non-tcam options (density change) Reduced counters, QoS, queueing complexity BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Buffering Ports On-chip limited to microseconds But Off-chip requires ASIC I/O pins fewer interfaces Board space and power Ports NPU Ports FIB Ports Commodity bandwidth is limited Many devices may be needed Better suited for buffering than FIB (1 read/write per packet optimized for 128B read/write) Stalled High-end graphics and networking moving to custom Custom memories Fewer devices are needed to reach bandwidth, fewer pins Development and per-unit costs Extremely expensive Ports Ports NPU FIB Ports Buffers BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 FIB Scale FIB On-chip FIB limited to ~256K LPM entries Usually an on-chip TCAM or lookup tree Can be scaled up to ~1M with well-known prefixes Combination of flat tables and LPM Ports NPU Buffers Ports Larger FIB tables require external memory: ASIC I/O pins redirected from interfaces to memory 4-8 reads for every packet high operations per second Board space (density) and power BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 System Scale Four common approaches FIB Fixed single NPU / Forwarding ASIC All resources dedicated to network interfaces Ports NPU Fabric May or may not have external memories Fixed multiple NPUs Buffers Connected via mesh or fabric chips Modular Expand with line cards Resources for fabric and usually external memories Multi-chassis Adds fiber connections to fabric cards (more power & board space) Increases software complexity BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Network Interface Features and Flexibility Packet Processing Engine (PPE) C programmable Run to completion Anything is possible, no pure optimization Wide range of pipeline programmability NCS 5000 < NCS 5500 < ASR 9000 < NCS6K CRS PPES Packet rate is strongly correlated to cost, power, and flexibility Counting stuff is much harder than most people realize Ingress Port Link Term Parser Layer Tunnel VLAN Service Trans Term Fwd PMF Egress NCS 5500 PIPELINE FEC Res FEC Res BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 IOS XR Platforms

27 The IOS XR Router Family CRS NCS 5000 NCS 5500 NCS 6000 ASR 9000 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 NCS 6000 First Dense 100G Highest router bandwidth capacity 128 Tbps first generation multi-chassis (2013) Only Tbps card on the market in With buffers and full FIB Dramatic improvement in power efficiency $6000/100G/month power savings over CRS-3 64x100 Merchant fabric Highly programmable forwarding, massive stats Custom memories for FIB and buffering Custom CPAK optics BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 NCS 6000 Line Card Architecture Slice architecture Optics, NPU & FIA per slice Run to Completion NPU w/ PPEs 1 Generation (so far) 1T line cards w/ 200G NPUs (40 nm) 400G NPUs in lab Optics Optics Optics T C A M Tables PPEs Packet Buffers npower X1 npower X1 T M Fabric Interface ASIC FIA TCAM for ACL/QoS scale Optics Optics npower X1 npower X1 FIA FIA Deep buffers (50+ msec) Optics npower X1 FIA DRAM CPU 10X 100GE BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 ASR 9000 Series Full range of chassis and interfaces 40G, 200G, and 1T generations Highly programmable forwarding Partnered for a customized NPU cisco software Highest scale, feature, and QoS capabilities Cisco fabric Commodity memories for FIB and buffering Mostly commodity optics BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 ASR 9000 System Design Backplane or midplane Flexible switch fabric options RP and fabric may be integrated into RSP Variable # of fabrics for increased capacity and redundancy First stage of fabric on line card Multiple airflow designs Front-to-back 9922, 9912, 9010 Side-to-back 9006 Side-to-side 9001, 9004 Modular options for lower-speed and legacy interfaces BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 ASR 9000 Line Card Architecture Slice architecture Optics, NPU & FIA per slice Flexible pipeline NPU 1 st stage of fabric on line card 3 Generations (so far) 40G line cards w/ 15G NPUs (90 nm) G line cards w/ 60G NPUs (55 nm) CPAK CPAK CPAK CPAK CPAK CPAK FIB TCAM NP-5c NP-5c NP-5c Packet Buffers FIA FIA FIA Fabric 800G-1.2T line cards with 240G NPUs (28 nm) TCAM for ACL/QoS scale CPAK CPAK NP-5c FIA Huge FIB (5M+) & buffers (200 msec) DRAM CPU 8X 100GE BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 ASR 9000 Modular Line Card Architecture PHY PHY 2/4/8X 10GE PHY PHY 20X 10G Bay 0 FIB TCAM NP-5c FIB TCAM Packet Buffers FIA Packet Buffers Fabric 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP 1G SFP PHY 40G QSFP+ 40G QSFP+ PHY 40G QSFP+ PHY 100G CPAK PHY 100G CPAK 100G CPAK PHY Bay 1 NP-5c FIA DRAM CPU 1G SFP 20X 1G 1/2X 40G 1/2X100G MODULAR 3 RD GENERATION LC BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 NCS rd generation 100G Dense 100GE with deep buffers 8 & 16-slot modular, 1 & 2 RU fixed Up to 57.6 Tbps in ½ rack in T - 3.6T line cards Options for route and ACL scale via TCAM Highly integrated Silicon Single ASIC for forwarding & fabric interface Dramatic power reduction to 0.24 W/Gbps QSFP28 optics GDDR5 commodity buffers BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 NCS 5500 System Design AIR INLET New chassis design for XR Orthogonal direct connect Horizontal line cards and vertical fabric Direct connection between line cards and fabric cards No midplane Distributed air intake between cards Fans cover fabric cards FAN REMOVED Provides cool air equally to all optics Optics require ~30C cooler operation than silicon Avoid preheating air from module to module REAR VIEW BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 NCS 5500 Line Card Architecture Slice architecture Optics & Pipelined Forwarding ASIC per slice integrated FIA Optional TCAM for FIB/ACL scale 1st Generation 3.6T line cards FA at 600G 2.9T line cards FA at 720G + TCAM Deep VoQ buffers (50+ msec) QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 Optics x 6 Optics x 6 Optics x 6 Optics x 6 Optics x 6 DRAM 36X 100GE Forwarding ASIC FA FA FA FA FA CPU Buffers BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP28 QSFP x 6 FA QSFP x 6 FA QSFP x 6 FA DRAM TCAM QSFP x 6 FA QSFP x 6 FA QSFP x 6 FA CPU Buffers SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ QSFP28 QSFP28 QSFP28 QSFP28 DRAM TCAM CPU Buffers NCS 5501/5502 Architecture Forwarding ASIC NCS RU Single 800 Gbps Forwarding ASIC 4x 100G + 40x 10G NCS RU 8 600Gbps Forwarding ASICs Integrated switch fabric 48x 100G Optional TCAM for scale Deep VoQ buffers (50+ msec) 40x10G NCS G Switch Forwarding ASIC Fabric QSFP x 6 FA 4x100G NCS T Switch 48x100G 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

38 NCS 5000 Platforms Extending IOS XR for satellite, ToR, and beyond Maximizing Silicon capabilities No external memories small FIB, usec buffers All bandwidth to ports, no fabric Low cost and power Full IOS XR Routing NCS 5001 NCS 5002 Forwarding ASIC 32x100 FIB Buffers NCS 5011 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 CRS-X Scaling CRS into 100G 4x 100G 40x 100G 2x 100G + 5x 40G 3x 100G + 1x 100G IPoDWDM Up to 51.2T via multi-chassis Fully compatible with CRS-1 & CRS-3 BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 Virtualized IOS XR IOS XR on x86 hardware with Linux VM & containers Router creation in seconds rapid service deployment Runs NCS 6000 software (recompiled) in emulated NPU Hosted & small PE with L2 & L3 VPNs Ideal route reflector 64 bit 10+ M routes BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 Virtualized IOS XR Performance & QoS Targeted for 5-50 Gbps forwarding 20+ Gbps with features per socket 3-layer Hierarchical QoS BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 One serious thing before we go... I m grateful for the opportunity to be here and to talk about something (at least I think) is interesting. It s important to me personally that attendees find these presentations helpful and/or valuable. Certainly not everyone will enjoy every presentation -- but you guys spend a lot of money to come here and Cisco and the speakers do a lot of work to put on the event and the presentations. If the material is excellent, we ve done our jobs. If there s something we can do better, please let us know... We really do read the comments. ;-) Thanks for your time today. Lawrence J Wobker, Cisco February 2017

43 Q & A

44 Cisco Spark Ask Questions, Get Answers, Continue the Experience Use Cisco Spark to communicate with the Speaker and fellow participants after the session Download the Cisco Spark app from itunes or Google Play 1. Go to the Cisco Live Berlin 2017 Mobile app 2. Find this session 3. Click the Spark button under Speakers in the session description 4. Enter the room, room name = BRKSPG Join the conversation! The Spark Room will be open for 2 weeks after Cisco Live Presentation ID 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 Complete Your Online Session Evaluation Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions BRKSPG Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47