High Performance File Serving with SMB3 and RDMA via SMB Direct

Transcription

1 High Performance File Serving with SMB3 and RDMA via SMB Direct Tom Talpey, Microsoft Greg Kramer, Microsoft

2 Protocol SMB Direct New protocol supporting SMB 3.0 over RDMA Minimal CPU overhead High bandwidth, low latency Fabric agnostic iwarp, InfiniBand, RoCE IP addressing IANA port (smbdirect 5445) File Client Application SMB3 Client Network w/ RDMA support R-NIC User Kernel Network w/ RDMA support R-NIC File Server SMB3 Server NTFS SCSI Disk

3 Documented MS-SMBD MS-SMB2 Windows krdma API NDKPI Part of Windows Driver Kit Network Direct (and Verbs) heritage

4 Implemented Windows Server 2012 SMB 3.0 over SMB Direct Supports Multichannel Continuous availability All other SMB 3.0 features

5 Basics SMB Direct is a transport framing Only 3 message types 2-way full duplex transport which supports: Datagram-type send/receive exchange With fragmentation/reassembly for large Direct RDMA Read/Write SMB 3.0 binding defines transport use: Client buffer advertisement for READ and WRITE Server RDMA buffer access (push/pull)

6 Use Discovery via SMB 3.0 Multichannel RDMA attribute of interface Negotiated capabilities SMB Direct version Message and RDMA Region sizes Credits Messages RDMA Read operations (via NDK provider)

7 Three messages SMB Direct Negotiate Request Octet 0 Octet 1 Octet 2 Octet 3 MinVersion MaxVersion Once Everything else SMB Direct Data Transfer Header Octet 0 Octet 1 Octet 2 Octet 3 Reserved CreditsRequested CreditsRequested CreditsGranted PreferredSendSize MaxReceiveSize MaxFragmentedReceiveSize SMB Direct Negotiate Response Octet 0 Octet 1 Octet 2 Octet 3 MinVersion MaxVersion Flags Reserved RemainingDataLength DataOffset NegotiatedVersion CreditsRequested Status Reserved CreditsGranted DataLength Padding Data (variable) MaxReadWriteSize PreferredSendSize MaxReceiveSize MaxFragmentedReceiveSize 7

8 Transfers Send/Receive model Single logical message Possibly sent as fragmentation train Using ordering properties of RDMA Implements crediting All SMB 3.0 operations use this Direct placement model Advertises RDMA regions in scatter/gather list SMB 3.0 uses for SMB2_READ and SMB2_WRITE Only. Piggyback on existing Channel

9 Send transfers DataOffset = 24 DataLength = 1000 RemainingDataLength = 1048 SMB Direct HDR (24 bytes) SMB3 message bytes Send 0 DataOffset = 24 DataLength = 1000 RemainingDataLength = 48 SMB Direct HDR (24 bytes) SMB3 message bytes Send 1 DataOffset = 24 DataLength = 48 RemainingDataLength = 0 SMB Direct HDR (24 bytes) SMB3 message bytes Send 2 9

10 SMB3 Reads and Writes SMB3 WRITE REQUEST Octet 0 Octet 1 Octet 2 Octet 3 StructureSize DataOffset Length Offset FileId Channel RemainingBytes WriteChannelInfoOffset WriteChannelInfoLength Flags Buffer (variable) Channel array SMB3 READ REQUEST Octet 0 Octet 1 Octet 2 Octet 3 StructureSize Padding Reserved Length Offset FileId MinimumCount Channel RemainingBytes ReadChannelInfoOffset ReadChannelInfoLength Flags Buffer (variable) Previously reserved fields Octet 0 Octet 1 Octet 2 Octet 3 Address Token Length 10

11 RDMA transfers SMB Direct READ SMB Direct HDR SMB3 HDR SMB3 READ REQ MEMORY DESCRIPTORS Send RDMA Write DATA Send SMB Direct HDR SMB3 HDR SMB3 READ RESP Client Server SMB Direct WRITE SMB Direct HDR SMB3 HDR SMB3 WRITE REQ MEMORY DESCRIPTORS Send DATA RDMA Read Send SMB Direct HDR SMB3 HDR SMB3 WRITE RESP 11

12 Credits Bi-directional Count of ready receive buffers offered Dynamic can increase or decrease at any time Optional to do so Used only to control low-level SMBD message exchanges Recycled independently of SMB operations Relatively small number required (100 s even for deep random workloads)

13 Quirks Interesting corner cases Last credit Always need 1 in each endpoint to avoid deadlock (but see details in spec!) Bi-directional no requirement for same both ways Async/Cancel/Errors No reply, multiple reply, unexpected large reply NOT an RPC-like interface, much as it may resemble one

14 Efficiency True bi-directional and streaming sends Can be exposed as sockets-like interface With register/unregister/rdma rw extensions RDMA operations / completions Datamover offload to RNIC Server pull model improves performance Many options for RDMA efficiency FRMR, silent completions, coalescing, etc Resources bounded by credits and sizes

15 Performance 15

16 SDC 2011 performance results 160,000 IOPS (1KiB random reads) RAID 0 12 s RAID 0 12 s 3200 MiB/sec (512KiB sequential reads) Single 32 Gbps InfiniBand link InfiniBand switch Nehalem: 1 socket x Ghz Westmere: 2 socket x Ghz 16

17 Current performance results File Client (SMB 3.0) SQLIO RDMA NIC RDMA NIC File Server (SMB 3.0) RDMA NIC RDMA NIC NTFS Storage Spaces HBA HBA JBOD JBOD

18 Current performance results sqlio2.exe -T100 t2 s60 b512 -o4 fsequential -BN LS (1 file per volume) Avg. MB/sec* Avg. IOs/sec (512 KiB) Avg. %CPU (Client) Avg. Latency (ms) 7,340 ~14K Server fully utilized sqlio2.exe -T100 -t16 s60 -b8 -o4 frandom -BN LS (four files per volume) Avg. MB/sec* Avg. IOs/sec (8 KiB) Avg. %CPU (Client) Avg. Latency (ms) 3,711 ~453K 60 < 1 Server fully utilized * 1MB = 1,000,000 bytes 18

19 Let s take it to 11! File Client (SMB 3.0) SQLIO RDMA NIC RDMA NIC RDMA NIC File Server (SMB 3.0) RDMA NIC RDMA NIC RDMA NIC NTFS Storage Spaces HBA HBA HBA HBA HBA HBA JBOD JBOD JBOD JBOD JBOD JBOD 19

20 Let s take it to 11 16! sqlio2.exe -T100 t2 s60 b512 -o4 fsequential -BN LS (1 file per volume) Avg. MB/sec* Avg. IOs/sec (512 KiB) Avg. %CPU (client) Avg. Latency (ms) 16,253 ~31K GigaBYTES (not bits) of storage throughput! * 1MB = 1,000,000 bytes 20

21 NUMA effects on performance At these speeds, NUMA effects cannot be ignored sqlio2.exe -T100 -t16 s60 -b8 -o4 frandom -BN LS (four files per volume) Test Case NUMA aware multichannel dispatcher NUMA unaware multichannel dispatcher Avg. MB/sec* Avg. IOs/sec (8 KiB) Avg. %CPU (client) Avg. Latency (ms) 3, K 60 < 1 3, K 76 < 1 To achieve peak performance, the SMB3 / SMB Direct stack must avoid cross-numa node memory accesses whenever possible. * 1MB = 1,000,000 bytes 21

22 NUMA and SMB3 Multichannel SMB3 Multichannel can be used to improve performance on NUMA systems SMB3 session is split across multiple channels Channels affinitized to a set of NUMA nodes Client dispatches IO requests to maximize performance and minimize cross NUMA node memory accesses One example of how the Windows Server 2012 SMB3 / SMB Direct stack has been optimized for high performance on NUMA systems 22

23 That s great! Now what? Are there simple improvements we could make to the SMB Direct protocol? Goals: Ease of implementation Increase IOPS Decrease latency Decrease CPU utilization 23

24 Where can we reduce IO costs? App SMB Client Client RNIC Server RNIC ReadFile() Aggressive invalidation: Consumes CPU cycles Consumes RNIC/bus cycles Increases interrupts/sec Increases IO latency ReadFile() status Register buffer Send SMB request Register status Send status Invalidate registration Invalidate status RDMA write data Send SMB response Consumes CPU cycles 24

25 Why aggressively invalidate? Application will likely reuse same buffers for subsequent IO requests. Why not cache and reuse buffer registrations? Peer can RDMA write after IO has completed Data corruption / system crash / connection loss Peer can RDMA read after IO has completed Data leak / connection loss Registration caches are not robust enough for storage and enterprise server applications. 25

26 Why aggressively invalidate? Invalidation provides strict correctness guarantees with respect to data: Data is in a consistent state following DMA Application can safely access its data Peer no longer has access to the region No data corruption, crashes, or leaks due to peerinitiated RDMA operations Aggressive invalidation is a necessary expense, but we might be able to reduce its cost 26

27 Use Send with Invalidate? App SMB Client Client RNIC Server RNIC ReadFile() Register buffer Send SMB request Register status Send status RDMA write data ReadFile() status Send SMB response with token to invalidate RNIC invalidates registration before indicating received data Consumes CPU cycles 27

28 Benefits of send with invalidate... Reduces RNIC work requests by 1/3 rd for small IOs (IOs that require one memory descriptor) Fewer CPU cycles Fewer RNIC/bus cycles Fewer interrupts Lower IO latency Already supported by major RDMA standards iwarp InfiniBand RoCE 28

29 Benefits of send with invalidate No change to SMB Direct protocol Make send with invalidate an optional feature. Client continues to invalidate the buffer if the server does not. Minimal change to SMB3 protocol SMB3 read/write request indicates when the server is requested to invalidate a request s memory descriptor via the server s response. Not a committed plan (investigation only) Feedback? 29

30 Summary SMB3 and SMB Direct allow Windows Server 2012 to efficiently host enterprise application workloads. SMB3 / SMB Direct protocols could be enhanced in simple ways to further improve performance. Increase IOPS Decrease CPU overhead Decrease latency 30

31 Questions? 31