from 1Gbps to 10Tbps Josef Ungerman CSE, CCIE#6167 Anatomy 2009 Cisco Systems, Inc. All rights reserved.

Transcription

1 Anatomy Anatomy of Core of Network Network Elements Elements from 1Gbps to 10Tbps Josef Ungerman CSE, CCIE#6167 1

2 Agenda 1. Basic Terms 2. Router Architectures 3. Switch Architectures 4. Hybrid Architectures 5. Network Processors 6. Switch abrics 2

3 Basic Terms Chapter 1 3

4 Cisco in 80 s: Router Architecture CPU DRAM lash, NVRAM, CON, AUX,... Packet interfaces Interconnect interfaces Store & orward Switching using packet buffers and QoS, handles WAN interfaces (very variable interface speeds) 4

5 Real-Time Packet Processing Process Switching CPU DRAM lash, NVRAM, CON, AUX,... process level interrupt level Packet interfaces Interconnect interfaces Process Switching Process handles the forwarding decision and other operations with the packet 5

6 Real-Time Packet Processing Data Plane vs. Control Plane CPU DRAM lash, NVRAM, CON, AUX,... process level interrupt level process region Control Packet I/O region Data Packet interfaces Interconnect interfaces Data Plane transit packets (aka. fast path) Control Plane packets for the router (routing, management, exceptions) routing/control plane = routing and vital functions (OSP, BGP, LDP, NTP, keepalives,...) management plane = access to the router (telnet, ssh, SNMP,...) 6

7 Real-Time Packet Processing (Network Processor) S/W vs. H/W router lash, NVRAM, CON, AUX,... (Network Processor) CPU u-code Route DRAM Control Packet Data Packet Packet DRAM Data Packet interfaces Interconnect interfaces (Network Processor) handles the data plane, not (platform-dependent) CPU runs handles only the control plane (platform-independent) Slow Path on the CPU can still forward some packets that cannot handle (eg. exceptions, non-ip protocols routing, unsupported features) 7

8 Real-Time Packet Processing (Network Processor) S/W vs. H/W router CPU Routing & orwarding Engine lash, NVRAM, CON, AUX,... Route DRAM Control Packet Data Packet (Network Processor) U header BQS Packet DRAM Data Packet interfaces Interconnect interfaces BQS (Buffering, Queuing, Scheduling) or TM (Traffic Manager) ASIC handles the memory access and QoS (packet body) U (Network Processing Unit) handles only packet forwarding and operations (packet header) 8

9 Summary what is inside the router? BBB basic building blocks Processor control-plane OS processor data-plane network processor Memory DRAM for OS memory and packet buffers SRAM for caches TCAM for fast lookups Interconnects bus serial link switch fabric We do not care about what is visible on the router chassis, fans, power supplies control ports CON, AUX, BITS, Alarms, Disks data ports LAN and WAN interfaces 9

10 It is always something (corollary). Good, ast, Cheap: Pick any two (you can t have all three). RC 1925 The Twelve Networking Truths 10

11 Column Mem. Column Mem. IM 10 IM 11 IM 12 IM 13 IM 14 IM 15 Column Mem. Column Mem. Packet Processing Technology Primer Performance vs. lexibility CPU (Central Processing Unit) multi-purpose processors (CISC, RISC) high s/w flexibility [weeks] low performance stability [cca 1Mpps today] usage example: access routers (ISR s) ASIC (Application Specific Integrated Circuit) mono-purpose hard-wired functionality low engineering flexibility [2 years] high performance stability [over 200 Mpps today] usage example: switches (Catalysts), core routers IM IM IM IM IM IM IM IM 2 6 IM IM 3 7 Input Demux Output Mux eedback (Network Processor) = something in between performance + programmability moderate s/w flexibility [months] moderate and stable performance [4Mpps 40 Mpps+] can be expensive, power-hungry, can have low code memory usage: fast feature-rich edge and aggregation 11

12 Memory Technology Primer Capacity vs. Access Speed Two basic memory technologies are in use today: Static RAM (SRAM, SSRAM) Dynamic RAM (DRAM, EDO DRAM, SDRAM, DDR) SRAM High Power High Speed [10-20ns] Low Density [eg. 16M per chip] DRAM Low Power Low Speed [40-60ns] High Density [eg. 1G per chip] 12

13 Interconnects Technology Primer Capacity vs. Complexity Bus half-duplex, shared medium for example PCI [800Mbps to 25Gbps+ today] simple and cheap Serial Lane (Point-to-Point Link Set) dedicated, unidirectional or full-duplex line for example SPI4.2 [11.2Gbps+ today] Switching abric (cross-bar, exchange) non-blocking, full-duplex, any-to-any for example GSR, CRS [40Gbps to 9.6Tbps+ today] 13

14 Example: Lookup Problem memory vs. processing TCAM (Ternary Content Addressable Memory) SRAM with a comparator at each cell ROOT 1 step very fast, but very expensive parallel, order independent lookups (ACL, QoS, Netflow, even IB) Content and Mask Address xxx xxx xxx Query 802 Result NEXTHOP Tree or Serial Lookup used by generic load share punt host-route cache drop glean incomplete used by the C12000, used by C10K memory vs. speed tradeoff! - could be (low SRAM) - could be used also for ACL, urp, accounting 14

15 Router Anatomy Chapter 2 15

16 undamental Building Blocks Module, card I/O module (hardware module with I/O interfaces Switch abric (any-to-any full-duplex switching element) Simplex serial link set Duplex serial link set Active and backup backplane connection (serial duplex link set) Bus Mux/demux, fabric interface (typically including a tiny buffer) Q orwarding ASIC (a complex of hardware elements and SRAM s handling data plane) Queuing ASIC (BQS Buffering/Queuing/Scheduling, TM Traffic Manager, etc.), Network Processor (programmable hardware element handling data plane) Packet buffering, packet memory, QoS point Control Plane element : CPU + DRAM + lash + NVRAM and control interfaces. CPU (Central Processing Unit) is a general purpose micro-processor running the OS (Operating System) 16

17 Cisco 7200 (1990 s) architecture Software Router data plane = interrupt level control plane = processes E-200 bridge I/O controller CON/AUX lash E PA PA PCI Bus 600 Mbps PA PA 1, 4 or 6 PA slots Bus L PCI Bus 600 Mbps Bus R PA PA 17

18 Cisco 7200 E-G1/G2 upgrade architecture Software Router no change, just faster CPU/memory E-G2 on-board 4x GE I/O controller PCI Bus 600 Mbps crypto CON/AUX/lash bridge PA PA PCI Bus 600 Mbps PA PA 1, 4 or 6 PA slots Bus L PCI Bus 600 Mbps Bus R PA PA 18

19 Cisco ESR10000 architecture Hardware Router data plane = PX chip (u-code) control plane = CPU with DMA chip for packet memory H/H 1.6G PRE (active) H/H SPA SIP-600 (2-slot) 11G Q SPA PRE (standby) Q ull Height Linecard 8 full-height slots (ESR 10008) 19

20 Cisco ASR1000 architecture Split data and control plane RP = control-plane only ESP = data-plane (QP chip) 20Gbps, 16Mpps, C-programmable SPA SPA SIP 11.2G ESP (active) Encryption Coprocessor ASR1006 RP (active) SPA SIP SPA ESP (standby) RP (standby) SPA SIP Encryption Coprocessor SPA 1-3 SIP slots 20

21 It is more complicated than you think. RC 1925 The Twelve Networking Truths 21

22 Cisco 7200: centralized single processor architecture Single-Processor one CPU for everything E-G2 on-board 4x GE CON/AUX/lash bridge I/O controller PCI Bus 600 Mbps PA VSA PA PA PCI Bus 600 Mbps PA PA 1, 4 or 6 PA slots Bus L PCI Bus 600 Mbps Bus R PA PA 22

23 Cisco 7500: distributed multi-processor architecture Multi-Processor distributed, parallel CPU s RSP (active) RSP (standby) memd memd VIP VIP PA PA PCI Bus 600 Mbps PA PA VIP PA PA Cy Bus 1Gbps Bus L Cy Bus 1Gbps Bus R VIP PA PA 3, 5 or 7 VIP slots 23

24 Cisco switch fabric architectural evolution Distributed orwarding Architecture up to 600Gbps today RP (active) RP (standby) Engine 0 Switch abric Cards arb. Engine M CSC redundant arb. CSC 10G Q Q SPA SPA Engine 2 Q Q 3G SC SC 40G Engine 6 Q Q 8, 12 or 16 Linecard/RP slots SC 24

25 midplane midplane Cisco CRS-1 architecture RP (active) Distributed orwarding Architecture up to 1.28Tbps today 8 multi-chassis: 10.24Tbps today RP (standby) SPA SPA SIP-800 MSC Q Q 49G 56G rx 98G 112G tx tx Switch abric Cards (all 8 active) Q P Q PLIM Q Q Q Q SPA SPA 4, 8 or 16 Linecard slots 25

26 midplane midplane Cisco CRS-1 evolution to 100GE Distributed orwarding Architecture up to 3.84Tbps single node in 2010 RP (active) RP (standby) SPA SPA SIP-900 MSC Q Q 140G rx 200G tx Switch abric Cards (all 8 active) Q P Q PLIM 100GE Q Q Q Q SPA SPA 4, 8 or 16 Linecard slots 26

27 Switch Anatomy Chapter 3 27

28 Switch vs. Router Definition 1 (old): switch is L2, router is L3 UNTIL PEOPLE INVENTED L3 SWITCH... Definition 2: switch is optimized for LAN E, GE, 10GE router is optimized for WAN many speeds 64Kbps 40Gbps UNTIL ETHERNET CAME TO WAN/CORE TOO... Definition 3: switch is optimized for speed (fixed functionality) router is optimized for functionality (programmable, modular) Programmable orwarding Engine Packet Buffers Queues and Shapers SWITCH NO shallow, max. ~16MB 4-8 per port ROUTER YES deep, hundreds of MB thousands per port Hardware Modularity 1-level 3-level+ 28

29 Catalyst 5000 (90 s) centralized switching architecture Sup (active) CatOS OS Bus linecard L2 Sup (standby) Bus linecard CatOS OS L2 Bus linecard Cybus 1Gbps 3, 5, 7, or 13 Linecard/Sup slots 29

30 Catalyst 5500 Netflow Switching: NC + RSM SupIII (active) CatOS OS Bus linecard L2 L3/4 NC (Netflow eature Card) RSM (Router/Switch Module) SupIII (standby) Bus linecard CatOS OS L2 L3/4 NC (Netflow eature Card) Bus linecard RSM (Router/Switch Module) 3x Cybus 1Gbps 3, 5, 7, or 13 Linecard/Sup slots 30

31 Catalyst 5500 Netflow Switching: NC + RSC SupIII (active) CatOS OS RSC (Route/Switch eature Card) Bus linecard L2 L3/4 NC (Netflow eature Card) SupIII (standby) Bus linecard CatOS OS RSC (Route/Switch eature Card) L2 L3/4 NC (Netflow eature Card) Bus linecard 3x Cybus 1Gbps 3, 5, 7, or 13 Linecard/Sup slots 31

32 Catalyst 6000 MLS (Multilayer Switching): PC + MSC Sup1A (active) Bus linecard CatOS OS L2/3/4 MSC (Multilayer Switching eature Card) PC (Policy eature Card) Sup1A (standby) Bus linecard CatOS OS L2/3/4 MSC (Multilayer Switching eature Card) PC (Policy eature Card) Bus linecard 3, 5, 7, or 13 Linecard/Sup slots D-Bus 16Gbsp R-Bus 4 Gbps EOBC bus 10Mbps 32

33 Real-Time Packet Processing ast-path orwarding (aka. Switching) CPU Routing Table Process CPU Routing Table Process flow setup IB setup ast Path low Switching (aka. ast Switching) the 1 st packet of a flow goes to slow-path the flow paths sets up the fast cache entry next packets of the flow take the fast-path ast Path IB Switching (aka. CE) RIB (Routing Information Base) IB (orwarding Information Base) RIB is mirrored to fast-path IB first, before any packets are switched traffic doesn t collide with control plane 33

34 Every old idea will be proposed again with a different name and a different presentation, regardless of whether it works. RC 1925 The Twelve Networking Truths 34

35 Catalyst 6500 Switch fabric Sup2 (active) Bus-only linecard SP RP MSC2 (Multilayer Switching eature Card) PC2 (Policy eature Card) SM active (Switch abric Module, 256Gbps) Sup2 (standby) abric-enabled linecard SP RP MSC2 (Multilayer Switching eature Card) PC2 (Policy eature Card) abric-enabled linecard + DC DC 8G 4, 6, 9, or 13 Linecard/Sup slots D-Bus 16Gbsp R-Bus 4 Gbps EOBC bus 10Mbps SM backup (Switch abric Module, 256Gbps) 35

36 Catalyst 6500 Sup720 Sup720 (active) Bus-only linecard SP RP MSC3 (Multilayer Switching eature Card) PC3 A,B,C (Policy eature Card) Integrated S 720 Gbps Sup720 (standby) abric-enabled linecard SP RP abric-only linecard abric-enabled linecard + DC 20G 4, 6, 9, or 13 Linecard/Sup slots D-Bus 16Gbsp R-Bus 4 Gbps EOBC bus 10Mbps 36 abric-only linecard

37 Nexus 7000 architecture RP (active) RP arb. CM RP (standby) RP CM P arb. P Switch abric Cards 48xGE 230 G 32xTGE 4, 8, or 16 Linecard slots X 37

38 Switch/Router Anatomy Chapter 4 38

39 midplane midplane Cisco CRS-1 extremely modular architecture RP (active) RP (standby) SPA SPA SIP-800 MSC Q Q Switch abric Cards (all 8 active) Q P40 Q PLIM Q Q Q Q SPA SPA 4, 8 or 16 Linecard slots 39

40 Cisco 7600 extremely compact architecture RSP (active) RSP (Route/Switch Processor) CPU + Switch-fabric active/standby S integrated ports + PC SP RP ES+40 ES20 RSP (standby) SP RP X6708 X6704 3, 4, 6, 9 or 13 Linecard/RSP slots 40

41 Cisco 7600 switch/router architecture X6700 WSM ACE RSP (active) SP RP SPA SPA SIP-600 ES+40 ES20 RSP (standby) SP RP X6708 X6704 SIP-400 SPA SPA 3, 4, 6, 9 or 13 Linecard/RSP slots Bus (headers only) 41 SPA SPA SIP-200

42 It is always possible to agglutinate multiple separate problems into a single complex interdependent solution. In most cases this is a bad idea. RC 1925 The Twelve Networking Truths 42

43 Cisco 7600 with ES+ new generation nice and clean RSP (active) SP RP 20G ES+ 20G ES+ RSP (standby) ES+ SP RP ES+ 3, 4, 6, 9 or 13 Linecard/RSP slots 43

44 Cisco ASR9000 router/switch architecture RSP (active) Transport LC 40G 45+45G Transport LC 80G RSP (fab. active) X 4 or 8 Linecard slots 44

45 Cisco ASR9000 double 100GE evolution RSP (active) 100G LC 200G LC G HGE HGE Existing LC 80G RSP (fab. active) Existing LC 40G X 4, 8 or 12 Linecard slots 45

46 Network Processors Chapter 5 46

47 orwarding ASIC s and Network Processors Price vs. Performance vs. lexibility Pipelining L2/L3 Switch ASIC example Catalyst Mpps, 320Gbps Parallel L2/L3 Switch ASIC example Catalyst Mpps, 80+Gbps Pipelining ASIC with u-program. stages example Cisco Mpps, 10Gbps complex features crtp, ATM, MVPN... Pipelining u-program. example Cisco 10000, 7600, Mpps, 18Gbps complex features PPPoE, ISG... Pipelining SMP (symmetric multiprocessing) u-program. example Cisco 7600, ASR Mpps, 20Gbps very complex features VPLS, PPPoE, ISG, Vidmon... Massive SMP u-program. example Cisco CRS-1 80Mpps, 40Gbps 185M transistors core/edge/eth. features Massive SMP C-program. example Cisco ASR Mpps, 20Gbps 1.3B transistors fully programmable in C >40 patents 100 engineers, 5 years sees full packet bodies! (firewall, IPSec, DPI, ISG,...) 47

48 Non-programmable L2/L3 Switching ASIC Catalyst 45/4900 L3 fwd police classify police orwarding Engine ASIC orwarding [Mpps]: 250 Throughput [Gbps]: 320 Programmability: None Netflow classify L2 fwd map queue TCAMs TCAM map statistics SRAMs headers only BQS ASIC - 2K queues - SRR (1L shaping) DRAM Linecards (8x 3Gbps fdx per LC) 48

49 One size never fits all. RC 1925 The Twelve Networking Truths 49

50 Pipelining Parallel Programmable ASIC GSR Engine5 (SIP-x01) DRAM DRAM TCAM SRAM orwarding [Mpps]: 16 Throughput [Gbps]: 10 Programmability: u-code u-code u-code L3 fwd CAM L2 fwd u-code ACL QOS N u-code RX or TX headers only etch u-code u-code L3 fwd CAM L2 fwd u-code u-code u-code L3 fwd CAM L2 fwd u-code ACL QOS N ACL QOS N u-code u-code Gather BQS ASIC - 8K queues - 2L shaping DRAM u-code u-code L3 fwd CAM L2 fwd u-code ACL QOS N u-code SRAM 50

51 u-programmable SMP U CRS SPP (Silicon Packet Processor) SRAM DRAM TCAM Resources Interconnect & Memory orwarding [Mpps]: 80 Throughput [Gbps]: 40 Programmability: u-code 185M transistors Processing Pool 188 Engines DRAM headers only Distribute/Gather BQS ASIC - 16K queues - MDRR 51

52 C-Programmable SMP U with complete packet processing ASR QP (Quantum low Processor) orwarding [Mpps]: 16 Throughput [Gbps]: 20 Programmability: C code 1.3B transistors (U = cca 500M) TI 90nm, 8L 0.51W per thread Processing Pool memory access 160 Engines (40 PPEs x 4 threads) on-chip resources SRAM DRAM 7 DRAM 0 TCAM complete packets Resources Interconnect & Memory complete packets Distribute/Gather BQS ASIC - 200K queues - 5L shaping HQ DRAM >100 engineers >5 years of development >40 patents 52

53 Switch abrics Chapter 6 53

54 Non-Blocking woo-doo RC1925: it is more complicated than you think! RC1925: the fast, the GOOD, and the cheap Ingress Linecards Egress Linecards TX 10G 10G RX Port-to-Port traffic zero packet loss TX 10G 10G RX TX 10G 10G RX Any-to-Any traffic Voice/Video/Data TX 10G 10G RX Unicast/Multicast non-zero packet loss! 54

55 abric Multicast Replication Egress vs. Ingress Replication Ingress Linecards TX TX RX RX RX Good: Egress Replication Cisco CRS-1, Cisco ASR9K, 7600 RX 10Gbps of multicast eats 10Gbps fabric bw! Ingress Linecards TX TX TX TX RX RX RX RX RX RX RX RX Good-enough: Binary Ingress Replication dumb switch fabric non-cisco TX TX TX TX RX RX RX RX RX RX RX RX 10Gbps of multicast eats 80Gbps fabric bw!! 55

56 abric QoS Head-of-Line Blocking and Arbitration Ingress Linecards TX 10G fabric arbiter 10G Egress Linecards RX Good abric QoS example GSR, ASR9000, Nexus TX 10G 10G RX per-destination queues strict priority for Voice/TV thousands of queues TX TX 10G 18.8G 10G 18.8G RX RX linecard arbiter Good-enough abric QoS example non-cisco traffic unaware arbiter (overspeed is needed) only 2 queues, no priority drops voice/video if loaded loss at >66% load 56

57 Marketing Math Ingress Linecard Egress Linecard Good: 40G 100G 40 Gbps per slot! non-blocking!!! 40G Network Processor 40G Network Processor Cisco CRS-1 16 slots: 16x16 matrix Good-enough: 40 Gbps per slot! non-blocking!!! 10G 18.8G 10G 18.8G 10G 18.8G 10G 18.8G 10G Network Processor 10G Network Processor non-cisco 12 slots: 48x48 matrix 57

58 Marketing Math Ingress Linecard Switch abric 3.84 Tbps Egress Linecard 100GE 120G Network Processor 140G 200G 120G Network Processor Cisco CRS planes 100GE?! No 100G processor available, internal loadbalancing across two old 50G processors 100GE??! No independent Switching abric available, if one CMP is removed, only 50Gbps throughput (no redundancy) Ingress Linecard loadbalancer 50G Network Processor Ingress Linecard 100G Network Processor 50G 50G 50G 50G Switch abric 1.6 Tbps 4:1 planes Active S Active S X Egress Linecard 100G Network Processor non-cisco 58 70G 70G 50G 50G Egress Linecard 50G Network Processor - abric slowdown! - Head-of-Line Blocking - additional latency - packet out of sequence non-cisco

59 errari = red car with horse 59

60 Conclusion The art of engineering optimization The ast, the Good, and the Cheap Router and Switch Architectures Centralized, Distributed, Hybrid Network Processors Programmable? Body visibility? Tunneling? Switch abrics Is 40G really 40G? Multicast Replication? Redundancy? QoS? 100GE? 60

61 61