Brent Callaghan Sun Microsystems, Inc. Sun Microsystems, Inc

Transcription

1 Brent Callaghan. Page 1 of 19

2 A Problem: Data Center Performance CPU 1 Gb Fibre Channel 100 MB/sec Storage Array CPU NFS 1 Gb Ethernet 50 MB/sec (via Gigaswift) NFS Server Page 2 of 19

3 Network vs Bus Performance Copy Copy Memory Bus Memory Bus NIC Network NIC Bus Memory Bus Memory Latency gets worse with each memory copy which further loads the CPU. Page 3 of 19

4 What is RDMA? DMA: Direct Memory Access RDMA: Remote Direct Memory Access Supports Direct Placement Networking offload for CPU Client Registered Memory Bus CPU RDMA READ/WRITE Server Registered Memory Bus CPU RDMA NIC RDMA NIC Page 4 of 19

5 RDMA Sweet Spot High bandwidth > 1Gb links Big chunks of data More than 1KB Short distance (low latency) 10 s of Meters Busy CPU Page 5 of 19

6 Protocols Using RDMA NFS/RDMA Benefits all RPC based applications Sockets Direct Protocol (SDP) Make RDMA accessible to sockets based apps SCSI RDMA Protocol (SRP) Needed for IB. Competition for iscsi? Direct Access File System (DAFS) Based on NFS version 4 DAFS Database Accelerator: TPC-C results are best price/performance for a UNIX based system. Page 6 of 19

7 RDMA Operations Registered memory Region referenced by handle, Address, length SEND srcaddr To Pre-posted Buffer Completion WRITE srcaddr, dstaddr Completion READ srcaddr, dstaddr Completion Page 7 of 19

8 RDMA Provides Direct Data Placement Header Header Header NIC NIC Header Data Data Buffer gather for transmit is easy RPC Message in contiguous network packet Direct data placement is hard without RDMA Page 8 of 19

9 Conventional RPC Data Movement XDR XDR Big chunk of data Page 9 of 19

10 RDMA RPC Data Movement SEND XDR XDR Big chunk of data READ or WRITE Direct placement with CPU offload Page 10 of 19

11 Small RPC Messages Client Server Most RPC Messages are small. Examples: LOOKUP GETATTR RPC Call RPC Reply RDMA SEND RDMA SEND Page 11 of 19

12 Big RPC Call Client Server Example: NFS WRITE RPC Call Data RDMA SEND RDMA READ Direct Placement RPC Reply RDMA SEND Page 12 of 19

13 Big RPC Reply Client Server Example: NFS READ RPC Call RPC Reply Direct Placement RDMA SEND RDMA SEND RDMA READ RDMA SEND Data Free buffers Page 13 of 19

14 Adapting NFS to RDMA NFS NFS NFS NLM ACL CHANGES RPC/XDR RDMA UDP TCP IP Page 14 of 19

15 Solaris Prototype Extension to Solaris kernel RPC Supports all NFS versions, 2, 3 & 4 All kernel RPC: Lock Manager, NFS_ACL Behaves like a normal NFS mount only a LOT more efficient! Supporting two RDMA flavors kvipl with Emulex GN9000/VI over Gigabit Ethernet File copy: 103 MB/sec vs 60 MB/sec over Gigaswift Infiniband Now testing with Mellanox Tavor cards Page 15 of 19

16 RDMA Flavors On Ethernet Emulex GN9000/VI - VI/TCP via 1Gb fibre RDMA Consortium RDMA over TCP IETF RDDP Working Group An interoperable, Internet standard Defined for SCTP and TCP On Infiniband Supported natively by all IB hardware Other Myrinet, Fibre channel, CLan,... Page 16 of 19

17 RDMA API Standardization Windows Sockets Direct Extensions to Winsock for RDMA Uses SDP UNIX Sockets Extensions Open Group s ICSC WG, Sockets API Extensions Direct Access Transport (DAT) Replaces VI API udapl & kdapl Interconnect Transport API (ICSC/Open Group) Based on DAPL Page 17 of 19

18 NFS/RDMA Standard NFS will continue to be an open, interoperable protocol! Need to publish standards for doing NFS/RDMA via Ethernet & Infiniband IETF has a role in RDMA protocol development RDDP Working Group IETF hosts standards for ONC RPC & NFS Proposal to extend NFSv4 charter to include NFS/RDMA protocol standard. Page 18 of 19

19 Page 19 of 19