Infiniband and RDMA Technology. Doug Ledford

Transcription

1 Infiniband and RDMA Technology Doug Ledford

2 Top 500 Supercomputers Nov 2005 #5 Sandia National Labs, 4500 machines, 9000 CPUs, 38TFlops, 1 big headache Performance great...but... Adding new machines problematic due to software interactions Diagnosing and locating faults very difficult

3 OpenFabrics Software Stack IP based App Application Level User APIs Diagnostic Tools Open SM Various MPIs Block Storage User Level MAD API SDP Library IPoIB to File Systems SDP User Level Verbs SRP iser Connection Manager (CM) Specific Verbs Common Verbs / API MAD Subnet Admin Client (SA Client) Connection Manager Abstraction (CMA) Core Clustered DB UDAPL Upper Layer Protocol Verbs / API Layer Sockets Based Headache Provider Hardware Specific Driver Hardware HCA RDS NFS-RDMA Cluster RPC FS iwarp Connection Manager (CM) iwarp Specific Verbs/API Hardware Specific Driver iwarp NIC Other FS

4 Throughput (MB/s) CPU Utilization (%) 0 10GigE w/ TOE 20Gig 0 Source: Head to TOE from OSU, and 10-Gigabit Ethernet for I/O in cluster computing from Sandia National Laboratories, and Mellanox

5 Wall Street Trading Environment Challenges Cancels per Trade 60.3 Quotes per Trade 63.6 CAGR 269% CAGR 72% * 2006 p 2005* = Aug 05 Source: NASDAQ * 2006 p 2005* = Aug 05 Source: NASDAQ Aggregated One Minute Peak MPS Rates CTS, CQS, OPRA, NQDS ,000 80,000 4,799 7,063 9,650 12,906 25,869 55, Feb 05 Jun 05 Dec 2005 Source: SIAC, OPRA, and NASDAQ Proj

6 Performance -Low Latency -High Bandwidth -Efficient CPU Utilization -Reliable Transport High Performance Computing High Performance Embedded Interconnects Storage Database Servers Middleware Servers Fibre Channel 10GigE Aggregation 1GigE Web Servers Enterprise LANs Number of Concurrent Applications

7 Price/Performance $69 (OEM) adapter IC vs. $500 for similar 10GigE adapter IC solution $200 (OEM) adapter card vs. $2000 for comparable 10GigE card 1.4GB/s and 2.7µs latency Virtualization Highest utilization of computing and storage resources Simplifies adding resources for rapidly expanding data centers Virtual Machine Hypervisor 20 Gb/s IB HCA GbE NIC

8

9 Infiniband/RDMA use climbing rapidly New Top500 Cluster New IB clusters New Myrinet clusters New Quadrics clusters

10 Level 1 IPoIB Application Level IP based App Sockets Based Various MPIs User APIs SDP Library UDAPL User Level Verbs Kernel Provided Interface IPoIB SDP Common Verbs / API Easiest to use, requires no modification of applications Lowest overall payback

11 Level 2 SDP Application Level IP based App Sockets Based Various MPIs User APIs SDP Library UDAPL User Level Verbs Kernel Provided Interface IPoIB SDP Common Verbs / API You might be able to use libsdp library to enable SDP in your application without any code changes or recompiles If not, the code changes to natively support SDP are very minimal This methods gets a good deal of the RDMA benefit

12 Level 3 Verbs/MPI Application Level IP based App Sockets Based Various MPIs User APIs SDP Library UDAPL User Level Verbs Kernel Provided Interface IPoIB SDP Common Verbs / API Code must be written to either the verbs or MPI API Code changes are not minimal, and in some cases require rethinking of application design This methods gets full benefit of RDMA capabilities

13 Key OpenFabrics Software Stack IP based App Application Level User Diagnostic Open SM Tools APIs User Level MAD API Upper Layer Protocol Verbs / API Layer Core Provider Hardware Sockets Based SDP Subnet Admin Client (SA Client) Hardware Specific Driver HCA Block Various Clustered Storage MPIs DB iwarp Specific * Future SDP Library User Level Verbs Connection Specific Manager (CM) Verbs MAD IB Specific UDAPL IPoIB SRP iser* Common Verbs / API Common RDS* iwarp Connection Manager (CM)* Connection Manager Abstraction (CMA) Hardware Specific Driver* iwarp NIC* iwarp Specific Verbs/API*