Comparing Ethernet and RapidIO

Transcription

1 Comparing Ethernet and RapidIO Greg Shippen System Architect Freescale Semiconductor The Embedded Fabric Choice Copyright 2007 RapidIO Trade Association Revision 1, Jan 2007

2 Agenda Interconnect Trends Technical Overview Comparison Applications Conclusion Copyright 2006 RapidIO Trade Association Slide 2

3 Market Trends Bandwidth Modularity, Reuse GB/s Protocols Cost Connected Devices CSIX TDM SPI4.2 PCI Express $$$ OPEX $$$ NRE TCP/IP ATM $$$ CAPEX Copyright 2006 RapidIO Trade Association Slide 3

4 Interconnect Trends Hierarchical Bus Bridged Hierarchy Broadcast PHY: Single-ended Shared Bus 1 st Generation Point-to-Point Single segment Broadcast Packet switched PHY: Single-ended Highest pin count PHY: Source-sync differential Lower pin count Example: HT/P-RapidIO Device Device Bridge Device Device Example: PCI/PCI-X/SCSI Device 133MHz Device Device Device Device Device Device Device Device Device Device Example: VME 2 nd Generation Point-to-Point 3 GHz Packet switched PHY: SERDES differential Lowest pin count Device Device Device Device 66MHz Device Device Performance Switch Fabric Device Device 10 GHz Ex: PCIe, Serial RapidIO, SATA, SAS Copyright 2006 RapidIO Trade Association Slide 4

5 Interconnect Roles Chip-to-chip Board-to-Device Board-to-board Chassis-to-chassis Chassis-to-chassis Device Chip-to-chip Board-to-Board Copyright 2006 RapidIO Trade Association Slide 5

6 Interconnect Clarifying Terminology Serial Interconnect Switched Interconnect Switched Fabric Link Protocol Point-topoint, no System Addressing Lightweight Serial Protocol System Addressing Ethernet I/O Load & Store Datagram (Packets) IO and Packets Protocols Robust Link Layer Traffic Managed CPU offload Copyright 2006 RapidIO Trade Association Slide 6

7 Agenda Interconnect Trends Technical Overview Comparison Applications Conclusion Copyright 2006 RapidIO Trade Association Slide 7

8 Ethernet Overview WAN scale interconnect Box-to-box, board-to-board, chip-to-chip, backplane Connect thousands to millions of endpoints Physical layer defined for LAN-scale interconnection Target market Closet to computer 100+ m distance Initially used in aggregation settings High performance switches, routers and LAN backbones Now WAN to workstations, PCs and laptops Gigabit Ethernet ubiquitous now Gigabit Specification standard completed in 1998 Gigabit Copper (1000Base-T) in 1999 No 10G Copper PHY defined yet Various MAC-to-PHY Standards defined Extensible Layered Specification Point-to-point packetized architecture Variable packet size High header overhead byte packet L2 PDU Up to 9000 byte jumbo frames End point DRAM Ethernet Endpoint Port 1 CPU MAC/PHY Port 2 Switch/ Router Port 0 End point Port 3 End point Copyright 2006 RapidIO Trade Association Slide 8

9 Ethernet Layer Preamble 4 Preamble Destination Address Destination Address Source Address Type/Length Packet PDU FCS Inter-Frame Gap Source Address Packet PDU SFD Bytes Layer 2 Packet Type: 1500 Byte Max Packet PDU Total = 294 Bytes (256 Byte PDU) Copyright 2006 RapidIO Trade Association Slide 9

10 Ethernet + UDP Preamble/SFD 8 Bytes L2 Header 14 Bytes IP Header 20 Bytes UDP Header User PDU FCS IFG 8 Bytes 256 Bytes 4 Bytes 12 Bytes 322 Bytes (256 Byte User PDU) UDP Packet Type: 1472 byte User PDU Copyright 2006 RapidIO Trade Association Slide 10

11 Ethernet + TCP/IP Preamble/SFD 8 Bytes L2 Header 14 Bytes IP Header 20 Bytes TCP Header User PDU FCS IPG 20 Bytes 256 Bytes 4 Bytes 12 Bytes 334 Bytes (256 Byte User PDU) TCP/IP Packet Type: 1460 Byte Max User PDU Copyright 2006 RapidIO Trade Association Slide 11

12 RapidIO Overview Chassis scale interconnect Chip-to-chip, Board-to-board via connector or cabling Physical layer defined for backplane interconnection ~ cm + 2 connectors (Serial) Target market Embedded systems Compute, defense, networking & telecom line cards CPU I/O, Line-card aggregation, backplane Serial PHY allowed expansion to data plane Flow control, encapsulation, streams Initially a processor interconnect First spec a Motorola & Mercury collaboration First revision standard completed in 1999 Rolled out with processors, bridges and switches Parallel 8-bit 500 MHz applied clock 5-6G PHY, 2, 8 and 16x lanes + Virtual Channels nearing completion Extensible Architecture Layered architecture Point-to-point packetized architecture Low overhead Variable packet size Maximum 256 byte PDU SAR support for 4 K-byte messages End point Host Port 2 Port 1 Switch Port 0 End point Port 3 End point Copyright 2006 RapidIO Trade Association Slide 12

13 RapidIO Packet Format: SWRITE AckID Prior tt 0 0 FTYPE ( ) Target ID Source ID 4 Address 0 XAdd 8 Packet PDU 80 Early CRC Packet PDU 88 Packet PDU CRC 266 Bytes (256 Byte PDU) SWRITE Packet Type: 256 Byte Max Packet PDU Copyright 2006 RapidIO Trade Association Slide 13

14 RapidIO Packet Format: Message AckID Prior tt 0 0 FTYPE ( ) Target ID Source ID 4 Msg length Source size Letter Mbox Msg seg Packet PDU 8 Packet PDU 80 Early CRC Packet PDU 88 Packet PDU CRC 268 Bytes (256 Byte PDU w/2 byte pad) Message Packet Type: 256 Byte Max Packet PDU, 4K w/sar Copyright 2006 RapidIO Trade Association Slide 14

15 Agenda Interconnect Trends Technical Overview Comparison Applications Conclusion Copyright 2006 RapidIO Trade Association Slide 15

16 Logical Layer Comparison Payload Size (Bytes) Memory-mapped R/W Layer (802.3) a (Jumbo) No Ethernet Layer (802.3) b (Jumbo) RDMA RapidIO Logical Layer Read/Write Atomics Configuration Write w/response No No Yes Address Size N/A 64-bits (RDMA) 34, 50, 66-bits Messaging Support No TCP 4KB user payloads, Doorbells Datagram Support Yes Yes 64KB user payloads c Globally Shared Memory No No Yes Deadlock Avoidance N/A L3+ must address Pervasive HW Support a Largest common jumbo frame size b Assumes 20-byte IP Header c Dataplane Extensions feature Copyright 2006 RapidIO Trade Association Slide 16

17 Transport Layer Comparison Ethernet RapidIO Layer 2 Layer 3+ Logical Layer Topologies Any Any Any Delivery Service Best Effort Guaranteed (TCP, SCTP, others) Guaranteed Best Effort a Routing MAC Address IP Address Device ID Maximum Endpoints (IPv4) (IPv6) 2 8 (Small) 2 16 (Large) Header Fields which change link-to-link None TTL, MAC Addr, FCS AckID Hop Count, CRC (Maint. Only) Redundant Link Support Yes Yes Yes a Data Streaming Spec Copyright 2006 RapidIO Trade Association Slide 17

18 Physical Layer Comparison Channel Data rate (Per pair) Symbol rate (Per pair) Encoding Signaling Signal Pairs (Per Direction) Electricals Clocking Gigabit Ethernet 1000Base-T 100m Cat5 Cable 1 Gbps 125 Mbaud 8b 4 quinary symbols 5 Layer PAM Code (Multilevel Signaling) 4 b Custom Embedded SERDES a ~50cm FR4 1 Gbps 1.25 Gbaud 8b 10b symbols NRZ 1 XAUI Embedded a Defacto standard. Some using 1000Base-CX electricals over backplanes b Each pair carries both Tx and Rx LP-LVDS ~50-80cm FR4 + 2 connectors Mbps Mbaud DDR NRZ 10, 19 LVDS Source Sync. RapidIO LP-Serial ~80-100cm FR4 + 2 Connectors 1, 2, 2.5 Gbps 1.25, 2.5, Gbaud 8b 10b symbols NRZ 1, 2, 4, 8, 16 XAUI (w/long & Short Reach) Embedded Copyright 2006 RapidIO Trade Association Slide 18

19 Protocol Efficiency 100% 90% 80% 70% 67% 80% 77% 93% 87% 80% 93% 89% 94% 96% 93% 97% 98% 93% 93% 99% Efficiency 60% 50% 40% 33% 50% 38% 63% 49% 66% RapidIO NWrite Ethernet L2 Ethernet UDP 30% 33% 20% 10% 0% 4% 1% 8% 2% 17% 5% 10% 19% PDU Size (Bytes) Copyright 2006 RapidIO Trade Association Slide 19

20 Effective Bandwidth Bandwidth (Gbps) SRIO 4x 3.125G SRIO 4x 2.5G SRIO 4x 1.125G 10G Ethernet: UDP Includes 1G Ethernet: 8B/10B, UDP header and ACK overhead when present PDU Size (Bytes) Copyright 2006 RapidIO Trade Association Slide 20

21 Quality-of-Service (QoS) Dependencies QoS depends on proper hooks across the interconnect fabric Hierarchical Flow Control Addresses short, medium and long-term congestion events Link and end-to-end Ability to define many streams of traffic Often defined as a logical sequence of transactions between two endpoints Ability to differentiate classes of traffic among streams Ability to reserve and allocate bandwidth to streams and classes Overall Interconnect Traffic Streams Classes Copyright 2006 RapidIO Trade Association Slide 21

22 QoS Comparison: Ethernet No universal QoS standard Some Layer 2+ switches support Priority Tagging (802.1d/q) Eight classes Increasing number of routers support MPLS at L Bytes PRIO CFI VID TCI Preamble/SFD L2 Header IP Header 8 Bytes 14 Bytes 20 Bytes UDP Header User PDU FCS IFG 8 Bytes 256 Bytes 4 Bytes 12 Bytes UDP Packet Type: 1472 byte User PDU 324 Bytes (256 Byte User PDU) Copyright 2006 RapidIO Trade Association Slide 22

23 QoS Comparison: RapidIO All implementations must support 3 prioritized flows No ordering between flows Allows shared buffer pool across flows Switches required to provide some improved service Extent of improvement is implementation dependant Dataplane Extensions adds carrier-grade QoS Support for thousands of flows, hundreds of traffic classes End-to-end traffic management End Point W Flow 2 End Point Y Switch End Point Z Flow 0 End Point X Flow 1 Copyright 2006 RapidIO Trade Association Slide 23

24 Flow Control Comparison Ethernet RapidIO Link-to-link flow control Link-to-link flow control PAUSE frames Congestion control L3+ end-to-end flow control XON, XOFF ECN, TCP windowing, others End point End-to-End Traffic Mgmt XOn-XOff Congestion Control End point Fine-grained end-to-end flow control Data Streaming Logical Layer End point Switch Switch Switch End point Link-level Flow Control Line Card End point Back Plane End point Line Card Copyright 2006 RapidIO Trade Association Slide 24

25 Software/Hardware Interface Ethernet PCI Express: Rd, Wr TCP/IP UDP Layer 2 RapidIO: SWRITE, MSG, Streaming Client SW Application Application API (Sockets?) Application App SW Stack SW Client Driver SAR/Err SAR/Err DMA Stack SW HW TCP/IP DMA UDP/IP DMA DMA HW Logical Trans MAC MAC MAC Link/PHY High-bandwidth interconnects require low CPU overhead usage model Hardware support for logical, transport and link layer Low overhead DMA with QoS support Copyright 2006 RapidIO Trade Association Slide 25

26 Ethernet Performance Microsecond+ fall through latencies (~100us?) Not just the hardware, data has to traverse the SW stack High CPU overhead Rule of thumb appears to be borne out in data for TCP/IP SW overhead 1 Hz of CPU per bit of throughput (per direction) Wire speed achievable with GHz class processors Some CPU will be left but how much depends on Protocol being terminated Offload features of GigE interfaces Too often advanced off-load features cannot be leveraged OS & SW stack support issues UDP or MAC/Layer 2 solutions sometimes also use proprietary higher layer protocols Can defeat the value of off-the-shelf standards-based solution Error correction at endpoint stacks introduce latency jitter and determinism issues Works well for application bandwidth < ~300Mbps Lack of flow control problematic for systems that cant significantly overprovision Copyright 2006 RapidIO Trade Association Slide 26

27 RapidIO Performance Latency Sub-microsecond switch latencies End-to-end latency Lower latency than Ethernet since latency does not include a SW stack Architecture RapidIO switches straightforward and orthogonal in architecture Strict peer-to-peer Packet headers architected to reduce logic No need to recalculate CRC Copyright 2006 RapidIO Trade Association Slide 27

28 Some Economics RapidIO and Ethernet with modest TCP/IP offload have similar underlying silicon costs Aggressive TCP/IP Offload engine larger than a RapidIO endpoints GigE Copper PHY is very large (~20mm2 in 130nm) Leveraging Ethernet volume economics not always a reality L2+ Ethernet switches suitable for aggregation and backplanes are not high volume ports, QoS features and SERDES PHYs for backplane ports, QoS features for aggregation Terminating TCP/IP demands significant processor overhead Dedicate processor or reduce performance and/or application features Copyright 2006 RapidIO Trade Association Slide 28

29 Agenda Interconnect Trends Technical Overview Comparison Applications Conclusion Copyright 2006 RapidIO Trade Association Slide 29

30 Summary Ethernet widely used in low bandwidth embedded applications Undisputed standard for wide area networks (WAN) 10G Ethernet role in backplane is yet to be determined Broad endpoint silicon and software support Flexible and adaptable software protocol stack High overhead & latency Significant cost jump for bandwidth above 1Gbps No standard Off-load or backplane SERDES PHY RapidIO has captured DSP and line card aggregation role Looking to expand role onto the backplane Low overhead protocol supporting both control and data plane Superior Quality-of-Service Variety of PHY speeds Cost competitive against 1G and 10G Ethernet Growing ecosystem Copyright 2006 RapidIO Trade Association Slide 30

31 Application Fit By Bandwidth Application Ethernet RapidIO Interconnect Bandwidth < 300 Mbps Control Plane (Low latency, Reliable) Stack Latency Data Plane (QoS, Streams, HA) 300 Mbps < Bandwidth < 1 Gbps Control Plane Bonding, 10Ge Data Plane Bonding, 10Ge 1 Gbps < Bandwidth < 10 Gbps Control Plane Bonding, 10Ge Data Plane Bonding, 10Ge Bandwidth > 10 Gbps Control Plane 10Ge Bonding Data Plane 10Ge Bonding PRIO 16-bit PRIO 16-bit Good Fit Marginal Poor Fit Copyright 2006 RapidIO Trade Association Slide 31

32 Application Fit By Attribute Low latency Low CPU overhead High Availability System Requirement High backplane bandwidth (310 Gbps+) High QoS (low latency jitter, many streams, flow control) Ethernet Requires CPUs Stack Traversal SW Stack Flow Control RapidIO Distributed computing 5+ endpoints in a network High Latency Good Fit Marginal Poor Fit Copyright 2006 RapidIO Trade Association Slide 32

33 Multiservice Switch SONET MultiPHY Line Card OC-12/ STM-4 Optics Optics Optics Optics MPHY Framer Packet Processing Control Processor Switch GbE Gigabit Ethernet Line Card PHY PHY PHY PHY Packet Processing Control Processor Switch RapidIO Backplane Fabric Fabric T1/E1 Voice Line Card LIU LIU LIU LIU LIU LIU LIU LIU Control Processor DSP DSP DSP DSP Switch DSP DSP DSP DSP Copyright 2006 RapidIO Trade Association Slide 33

34 Mobile Switching Center SDRAM Flash Flash From BSC/RNC Host Processor Host Interface: Initialization Software Download Configuration Signaling Support Network Management DSP DSP DSP DSP DSP Ethernet/ ATM Aggregator Logic DSP Serial RapidIO Switch DSP DSP DSP DSP Backplane TDM Bus To PSTN Copyright 2006 RapidIO Trade Association Slide 34

35 DSLAM Voice Gateway Module SDRAM SDRAM Control Processor Flash Flash Serial RapidIO to Backplane Backplane Octal Framers T1/E1 Ports TSI TSI Serial RapidIO Switch Octal Framers DSP DSP DSP DSP DSP DSP DSP DSP DSP DSP Copyright 2006 RapidIO Trade Association Slide 35

36 Enterprise Storage Switch To Media: Fibre Channel, SCSI Disk Disk Cache Cache Memory Media Interface Media Interface Memory board... CPU CPU CPU CPU Media Interface board... RapidIO Backplane... CPU CPU CPU CPU Disk Disk Cache Cache Memory Memory board... Media Interface Media Interface Server Interface board... To Servers: (FibreChannel, GbE) or mainframes (ESCON, FICON). Media Interface CPU CPU CPU CPU Media Interface Media Interface CPU CPU Media Interface board CPU CPU Media Interface Server Interface board Copyright 2006 RapidIO Trade Association Slide 36

37 Signal and Image Processing RapidIO Backplane XBAR XBAR Node ASIC DDR SDRAM Node ASIC DDR SDRAM Node ASIC DDR SDRAM Node ASIC DDR SDRAM Node ASIC DDR SDRAM CPU EEPROM CPU EEPROM CPU EEPROM CPU EEPROM CPU EEPROM Copyright 2006 RapidIO Trade Association Slide 37

38 Conclusion Gigabit Ethernet will serve a limited role as a system interconnect Works in low performance settings where significant over provisioning is possible RapidIO will expand its existing role as the standard system fabric Available now Efficient protocol supporting both control and data plane Variety of PHY speeds Cost competitive underlying economics Copyright 2006 RapidIO Trade Association Slide 38