Low Latency Server Virtualization Using RapidIO November 29-30, 2011 Server Design Summit Mohammad Akhter Devashish Paul Integrated Device Technology 2010 Integrated Device Technology, Inc.
The Analog and Digital Company The World Leader in Timing, Serial Switching and Interfaces in Silicon Timing Analog in Memory Interfaces System in DisplayPort Expertise in RapidIO Switching in PCIe Solutions in Business PC Audio in VME Bus Solutions in PowerPC Host Analog Bridges Application-Optimized Mixed-Signal Solutions Maximum system performance Lowest overall Bill-of-Materials Shortest time-to-market Communications Computing Consumer Re-thinking the System to Architect a better solution PAGE 2
RapidIO Success RapidIO Enables Disruptive Server Architecture Server IO Virtualization IO Virtualization Flows Servers with RapidIO IO Virtualization using RapidIO Summary Agenda PAGE 3
Lowest Latency Interconnect ~ 100 ns Shipped millions of devices Scalable System RapidIO Success Inherently scales to large system with thousands of nodes Scale s, blades, Aggregate Chassis to large systems Addresses Critical Interconnect Problems in multiple markets Military, Video, Computing, Wireless Industry Large Eco-System Silicon Partners and Suppliers PAGE 4
RapidIO Enables Disruptive Server Architectures Supports both x86 and ARM based Architectures Eliminates need for large ToR Switches Large Servers x86 CPU with RapidIO Micro-servers using ARM CPU with power efficient low cost RapidIO Interconnect Supports Thousands of s and flows Best-in-class end-to-end latency Switch latency around 100 ns Supports Secured Virtualization Supports any kind of topology High Performance Low Latency RapidIO based Server Start, Mesh, Dual-star, Hypercube, Torus etc.. RapidIO Switch Hub RapidIO Switch RapidIO CPS-1848 48 lane switch PCIe2 S-RIO 2 PCIe2 S-RIO 2 PCIe2 S-RIO 2 PCIe2 S-RIO 2 x86 x86 x86 x86 CPU CPU CPU CPU RapidIO Based RapidIO CPS-1848 48 lane switch PAGE 5
RapidIO in Server Lowest latency 100 ns Scales to 1000 s of nodes 8W per switch < 2 W per NIC 13x13 mm PCIe to S-RIO NIC RapidIO Switch Hub RapidIO CPS-1848 48 lane switch RapidIO Switch PCIe2 S-RIO 2 PCIe2 S-RIO 2 PCIe2 S-RIO 2 PCIe2 S-RIO 2 x86 CPU x86 CPU x86 CPU RapidIO Based x86 CPU RapidIO CPS-1848 48 lane switch 13x13 mm PCIe to S-RIO NIC For < 2W PAGE 6
Server IO Virtualization What is Server IO Virtualization Logical sharing of physical IO resources between server and data center resources Why care? On-demand efficient allocation of IO Match CPU and OS processing requirements to IO Allow better utilization of CPU Allow many nodes in the system to access IO resources PAGE 7
Server IO Virtualization Applications OS Kernel/Driver Driver/OS ory ory Memo CPU 0 Hardware I/O Device PAGE 8
Server IO Virtualization Applications (Kernel/Driver) Driver/OS Mem Mem mory CPU 0 Hardware I/O Device PAGE 9
IO Virtualization - Flows Memo mory (Kernel/Driver) (Kernel/Driver) CPU 0 Memo Memo ory CPU 0 I/O Device I/O Device SAN LAN PAGE 10
IO Virtualization - Bypass Memory (Kernel/Driver) CPU 0 I/O Device Communication through is expensive Higher CPU overhead Higher latency Early days IO virtualization was done entirely by software Virtualized enabled IO Card bypasses Data path goes directly from OS to IO Card IO device creates virtualized view of distinct resources and address space to OS Control/Management functions may still flow through PAGE 11
IO Virtualization using RDMA (Kernel/Driver) (Kernel/Driver) Memory Memory CPU 0 RDMAP DDP MPA TCP Memory CPU 0 I/O Device I/O Device Ethernet Switch Today s I/O NIC have too many steps for RDMA! Higher latency and Power Higher cost and CPU overhead PAGE 12
IO Virtualization using RapidIO (Kernel/Driver) (Kernel/Driver) Memory Memory CPU 0 Memory CPU 0 Tsi721 I/O Device Tsi721 I/O Device S-RIO Switch PCIe to S-RIO bridge and Switch Simplies IO Virtualization Improves RDMA without overhead built in DMA/Messaging Reduces latency Supports large number of nodes Improves IO and CPU utilization PAGE 13
RapidIO: Low power Servers (Kernel/Driver) (Kernel/Driver) Memory Memory CPU 0 Memory CPU 0 Tsi721 I/O Device Tsi721 I/O Device S-RIO Switch PCIe to S-RIO bridge replaces your NIC for <2W in a 13x13 package!! PAGE 14
RapidIO: Layered Protocol System Integration Implem mented in Ha ardware Logical Specification Transport Specification Physical Specification Logical I/O x8, x16 Parallel Message Passing Globally ll Shared Memory Transport x1, x2, x4, x8, x16 Serial Flow Control Data Streaming Multicast Future Physical Specs System Bring up Interop Error Mgmt PAGE 15
IO Virtualization using RapidIO Low latency Memory to Memory cut-through transactions Bypass for data transfer I/O virtualization in RapidIO 1000 s ofs with 16 bit deviceid Message Passing Logical Layer Map Mailbox, Letter, xmbox to IOV needs Direct IO Logical Layer Many nodes with 66-bit source/destination address and TID Data Streaming Logical Layer VM VNIC VM VNIC VMM VMM API Management Device Driver 1000 s of flows with streamid, xtype, xh CPU 0 Memory Memory RapidIO PAGE 16
x86 Server with RapidIO Gen2 (Today) PCIe to S-RIO bridge and Gen2 S-RIO Switch brings x86 CPU into RapidIO based Servers Multiple p links each 20G Multiple storage, networking, computing and chassis links Scalable Virtualized System PAGE 17
Native RapidIO Gen2 in ARM Server Native RapidIO with ARM RapidIO Gen2 Switch Multiple l links each 20G Multiple storage links Multiple compute node links Multiple networking links Lowest power Lowest latency Lowest footprint Secured I/O Virtualization ti with RDMA PAGE 18
x86 Server with RapidIO 10xN (Future) 10xN RapidIO-PCIe Bridge brings x86 into RapidIO based Servers Support Large Servers Live VM migration Lowest Switch Latency Modular Scalable Architecture Upto 160 Gbaud per port with 10.3125 G per lane PAGE 19
ARM Server with Native S-RIO 10xN Native on-chip RapidIO Interconnect Lowest Latency Lowest Power consumption Smallest Footprint Secured Virtualized Architecture PAGE 20
Example S-RIO based Server Cluster with RapidIO NIC Scalable Architecture t with RapidIO Switch and NIC Any kind of Topology PAGE 21
Summary RapidIOR Simplifies Virtualization ti Low overhead RDMA Live Virtual Machine migration 1000 s of nodes and flows 100 s of virtual NICs Lowest Latency Interconnect in the industry ~100 ns switch latency Green Solution 8W per switch and <2W per NIC Enables large Data Center for Cloud computing Low Power Micro servers with ARM and NATIVE RapidIO Macro servers with x86 and PCIe to RapidIO NIC RapidIO Simplifies IO Virtualization Lowest Latency, Power, and Cost PAGE 22
Visit Serial RapidIO Products at www.idt.com Product Data Sheets Reference Designs RapidIO Trade Association http://www.rapidio.org/home/news/membernews RapidIO Overview http://www.eetimes.com/design/signal-processing- g p g dsp/4017736/backplane-tutorial-rapidio-pcie-and-ethernet RapidIO versus Ethernet http://advancedtca-systems.com/comparing-ethernet-rapidio/ Contact Integrated Device Technology 6024 Silver Creek Valley Road San Jose, CA 95138 PAGE 23