OFED Storage Protocols

Transcription

1 OFED Storage Protocols R. Pearson System Fabric Works, Inc.

2 Agenda Why OFED Storage Introduction to OFED Storage Protocols OFED Storage Protocol Update 2

3 Why OFED Storage 3

4 Goals of I/O Consolidation Cluster of Servers Multiple Fabrics High CapEx and High TCO Single InfiniBand Fabric Low CapEx and Optimal TCO One Wire 4

5 Unified InfiniBand Fabric Performance Ultimate scalability Cost of ownership Dedicated I/O channels enable consolidation 5

6 Industry Support for Native InfiniBand Storage 6

7 Storage Interconnects Type Max length Peak data rate Max devices PATA (IDE) 46cm 133MB/s 2 SATA/eSATA 1m/2m 300MB/s 1 SCSI 12m 320MB/s 16 SAS 6m 300MB/s 16K FC 50km 400MB/s 16M iscsi (Ethernet)??? IB Cu (DDR) 15m/5m 1gbps/2gbps 16M(48K?) IB optical (mm/wan) ~300m/? 1GB/s 16M 7

8 Introduction to OFED Storage Protocols 8

9 Linux Storage Stack (das/san) User Kernel Application VFS Layer (read, write, open, close, ) File Systems (ext2, ext3, reiser, xfs, ) Block Drivers SCSI Mid Layer HBA Drivers (sd, md, lvm,..) (scsi, fc, sata, ide, ) Storage HBA 9

10 Linux Storage Stack (nas) User Kernel Application VFS Layer NFSD (cache) (read, write, open, close, ) RPC Filer TCP/IP NIC Driver NIC 10

11 Linux Storage Stack (iscsi) User Kernel Application VFS Layer File Systems Block Drivers SCSI Mid Layer iscsi TCP/IP NIC Driver Storage NIC 11

12 OpenFabrics Software Stack Application Level IP Based App Access Sockets Based Access (IBM DB2) Various MPIs Block Storage Access Clustered DB Access (Oracle 10g RAC) Access to File Systems SA MAD Subnet Administrator Management Datagram User APIs Upper Layer Protocol Diag Tools Open SM User Level MAD API SDP Library UDAPL User Level Verbs / API IPoIB SDP SRP iser RDS NFS-RDMA RPC User Space Kernel Space Cluster File Sys SMA PMA IPoIB SDP SRP Subnet Manager Agent Performance Manager Agent IP over InfiniBand Sockets Direct Protocol SCSI RDMA Protocol (Initiator) Mid-Layer SA Client MAD SMA Connection Manager Connection Manager Abstraction (CMA) Connection Manager iser RDS UDAPL iscsi RDMA Protocol (Initiator) Reliable Datagram Service User Direct Access Programming Lib InfiniBand Verbs / API R-NIC Driver API HCA Host Channel Adapter Provider Hardware Hardware Specific Driver InfiniBand HCA Hardware Specific Driver iwarp R-NICR R-NIC Key RDMA NIC Common InfiniBand iwarp Apps & Access Methods for using OF Stack 12

13 Linux Storage Stack (das/san) User Kernel Application VFS Layer (read, write, open, close, ) File Systems (ext2, ext3, reiser, xfs, ) Block Drivers SCSI Mid Layer SRP/iSER (sd, md, lvm,..) Storage OFED HCA 13

14 Linux Storage Stack (nas) User Kernel Application VFS Layer NFSD (cache) (read, write, open, close, ) RPC Filer TCP/IP IPoIB OFED HCA 14

15 Linux Storage Stack (iscsi) User Kernel Application VFS Layer File Systems Block Drivers SCSI Mid Layer iscsi TCP/IP IPoIB OFED Storage NIC 15

16 Common I/O Scheme Initiator Fabric Target System Memory CPU CPU command Storage hardware Logical View Physical View RDMA read/write Buffers response 16

17 Common I/O Scheme Initiator Fabric Target System Memory CPU CPU command rkey lkey Storage hardware MR Logical View Physical View RDMA read/write Buffers MR response 17

18 Common I/O Scheme Initiator Fabric Target System Memory CPU CPU command rkey Storage hardware lkey Logical View Physical View RDMA read/write Buffers response 18

19 Common I/O Scheme Initiator Fabric Target System Memory CPU CPU command rkey lkey Storage hardware MR Logical View Physical View RDMA read/write Buffers FMR response 19

20 OFED Storage Protocol Update 20

21 SRP Update SCSI RDMA Protocol (SCSI over IB) T10.org specification, similar to FCP (SCSI over FC) All data transfers use InifiniBand RDMA Initiator Linux proprietary and open source Gen1 -based in production OFED 1.0 SRP Initiator in Beta (still in development) Windows Beta release in July Target Linux open source Gen1 -based target on OFA Vendors in production or in final qualification Linux and other RTOSs 21

22 SRP Next Steps Initiator Host failover/multipathing Boot support for Linux and Windows Target Call for interest to port target to OFED Mixed file system / block storage Linux targets require this capability 22

23 iser Update iscsi Extensions for RDMA IETF IP storage specification, Leverages iscsi management and discovery protocols iser eliminates iscsi/tcp bottlenecks RDMA, hardware transport Initiator Production supported in OFED 1.0 Tested against Voltaire iser target running in IB/FC router Target Linux open source Gen1 -based target on OFA Vendors porting to OFED 23

24 NFS RDMA Status Release 5 NFS/RDMA client and server for Linux on SourceForge Supports OFA CM and RDMA verbs API Client maintainer is Tom Talpey, Netapp Server maintainer is Tom Tucker, Open Grid Computing Linux NFS RDMA client and server testing Passes the Connectathon test suite, IOzone and network stress tests Tested with several Mellanox-based IB HCAs and standard Linux x86 servers Initial test data shows 800MB/s+ read to cache Release 5 ready for evaluation by users, system integrators and OEMs 24

25 OFED Storage Summary Benefits of consolidated InfiniBand I/O are clear and being realized by real-world deployment today Vendors in production or in final qualification stages of block storage solutions SRP and iser will both continue development in OF NFSoRDMA ready for community trials Stay in lock step with OFED Vendors porting clustered file systems to OFED 25

26 System Fabric Works Software/architecture/consulting services Vendor neutral Experienced team Customers include: DDN/LSI/Chelsio/QLogic/Cisco/ Silver Storm/Trilabs/Agilent/USDoD/GE/Panta/ Themis 26

27 Embedded Products Vendor Part Number Description Diversified Technology ATC5232 IB ATCA SBC ATC2148 ATCA IB Switch SBS/GE Fanuc IB4X-CPCI-2A 2 port HCA cpci 3U/6U Sky Computers AMS0508 AMS1412 IB4X-PMC-2A TDB TDB Various 5 slot ATCA chassis IB compatible 14 slot ATCA chassis IB compatible 2 port HCA PMC module VITA41 IB switch VITA41 IB SBC VxWorks 5.5 IB stack IB based systems 27