SMB 3.0 Performance Dan Lovinger Principal Architect Microsoft

Transcription

1 SMB 3.0 Performance Dan Lovinger Principal Architect Microsoft

2 Overview Stats & Methods Scenario: OLTP Database Scenario: Cluster Motion SMB 3.0 Multi Channel Agenda: challenges during the development of SMB 3.0 during the development of Windows Server 2012 (and Windows 8)

3 Summary Statistics Then there is the man who drowned crossing a stream with an average depth of six inches. W.I.E. Gates (?) 3

4 Transactions/s Starting Point Metric v. Time 6000 Transaction Rate Time (s) All sorts of interesting things afoot!

5 Transactions/s Average Transaction Rate Average Time (s) Classic problem completely different behaviors have the same average understates steady state hides very interesting behavior Performance is often in the variation How often how consistently where are the outliers to analyze

6 # of Samples Distribution 60 Distribution : Transaction Rate Average in bucket Transactions/s Histogram / Distribution How often was the system running this fast? Note that data must be resampled into buckets In this example buckets are 20 transactions/s wide average located in the [ ) transactions/s bucket, with 21 samples

7 # of Samples Standard Deviation 60 Distribution : Transaction Rate σ Average +σ Transactions/s Familiar first approximation here, +/ transactions/s Strong meaning if the type of distribution is known Gaussian / Normal the classic Bell Curve - ~34% to each side which this is not Spans >90% of the total distribution, here

8 # of Samples Median Distribution : Transaction Rate Average Median Transactions/s The mid-point of all of the data points 50% higher 50% lower 50 th Percentile Coincidentally matches the peak of this distribution transactions/s

9 # of Samples Percentiles 60 Distribution : Transaction Rate Transactions/s Median Percentage of the dataset cut lines Relevant to guarantees of behavior / performance Median locates the center 10 s and 90 s No worse than / At least X, Y% of the time What makes sense for latency? Bandwidth?

10 # of Samples Percentiles Distribution : Transaction Rate Transactions/s -σ Median σ Looking back to standard deviation, it really didn t work. for a 30 minute real workload with a well defined steady state!

11 Percentile Cumulative Distributions 100% 90% 80% 70% Cumulative Distribution : Transaction Rate 60% 50% 40% 30% 20% 10% 0% Transactions/s Visualization of percentiles

12 Percentile OLTP 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Transaction Rate - Cumulative Distribution OLTP Simulation - Full Run Transactions/s SMB SMB+ DAS Note: for this section, SMB and SMB+ refer to a before and after change, not a revision of the protocol. The issue was not in SMB and very specific to workload. See summary. 12

13 OLTP OLTP OnLine Transaction Processing Log File small-midsize sequential IO Database File(s) small random IO Fundamental dependency on the SMB 3.0 feature set for continuously available connections to the database content New workloads new problems for an implementation

14 Simulated OLTP Hardware Configuration CPU Type Memory Amount Network Type Client Server 2 sockets with Ghz (12 cores total) 36 GiB Onboard 1GbE network interface Number of Network Links Storage Adapter Storage N/A N/A 1 x 1GbE 1 Fibre Channel Adapter 2x 4Gb/s connectivity 14x 10KRPM HDD RAID0 14

15 Simulated OLTP Hardware Configuration Client 1 Gb Ethernet 2x4Gb Fibre Channel Server DAS SMB 14 Disk RAID10 15

16 18 Years Later The Write Bubble NT Filesystem Lingo: Valid Data Length Offset in file to which user data has written data Allows efficient physical zeroing fully allocated files Code sharing from the FAT implementation May 1994 clever trick for VDL extension Nov 1994 ported over with a slight mislocation Causal requirements Simultaneous Read/Write, Async, NonCached IO to the same file No other workload affected! 16

17 Average Queue Length Simulated OLTP Average Disk Queue Length Distribution SMB SMB+ DAS Disk Queue Length 17

18 Percentile Simulated OLTP 100% 90% 80% 70% 8KiB Random IO/s Cumulative Distribution SMB v. DAS Windows 2008 R2 RTM 60% 50% 40% 30% SMB SMB+ DAS 20% 10% 0% IO Operations per Second 18

19 Samples Simulated OLTP SMB+ Client v. Server DAS Latency Windows 2008 R2 RTM Latency (ms) 500us Client Read Server Read Client Write Server Write Percentile Read Shift (%) Write Shift (%) Read Shift (us) Write Shift (us) 10% 13% 7% % 6% 4% % 3% 2% % 2% 1% % 1% 1%

20 Percentile Simulated OLTP 100% 90% 80% 70% 8KiB Random IO/s Cumulative Distribution SMB v. DAS Windows Server % 50% 40% Methodology Effect SMB SMB+ 30% DAS 20% 10% 0% IO Operations per Second 20

21 Samples Simulated OLTP SMB+ Client v. Server DAS Latency Windows Server Latency (ms) 410us Client Read Server Read Client Write Server Write Percentile Read Shift (%) Write Shift (%) Read Shift (us) Write Shift (us) 10% 12% 6% % 5% 4% % 2% 1% % 1% 0% % 0% 0%

22 OLTP Hardware Configuration CPU Type Memory Amount Network Type Client Server 2 sockets with Ghz (8 cores total) 24 GiB Onboard 1GbE network interface Number of Network Links Storage Adapter Storage N/A N/A 1 x 1GbE 1 Fibre Channel Adapter 2x 4Gb/s connectivity 24x 10KRPM HDD: 2x 10 HDD RAID0 (DB) 2 HDD RAID0 (LOG) 22

23 OLTP Hardware Configuration Client 1 Gb Ethernet 2x4Gb Fibre Channel Server DAS SMB 2x 10 Disk RAID0 (Data) 2 Disk RAID0 (Log) 23

24 Percentile OLTP 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Transaction Rate - Cumulative Distribution OLTP Simulation - Full Run Transactions/s SMB SMB+ DAS Full result is intuitively close Median 4270 transactions/s Mean 3920 transactions/s 4Gb v. 1Gb effect? 24

25 Transactions/s OLTP First Correction 7000 SMB v. DAS Transaction Rate - Ingest SMB+ Tps DAS Tps Time (seconds) 25

26 Million Bytes/s OLTP First Correction 300 SMB v. DAS Bandwidth - Ingest SMB+ Read Bytes DAS Read Bytes Time (seconds) 26

27 Percent of Total OLTP After First Correction Transaction Rate - Cumulative Distribution Post Ingest 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% SQL Transactions/s SMB+ DAS Far closer! 27

28 Transactions/s OLTP Second Correction 7000 Transaction Rate To Equal Work Steady State GB OLTP Log SMB+ DAS Time (seconds) 28

29 Million Bytes/s OLTP Second Correction 300 SMB v. DAS Transition Read Bandwidth To Equal Work Steady State GB OLTP Log Gb SMB+ DAS Time (seconds) 29

30 Percent of Total OLTP After Second Correction 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Transaction Rate - Cumulative Distribution Steady State - Work Matched SQL Transactions/s SMB+ DAS Again intuitive random IO bandwidth << 1Gb Only -3.0% performance relative to DAS over final 19 minutes of the run (+33s) 1Gb Ethernet very nearly meets 4Gb FC Log Volume Data Volumes DAS SMB2 DAS SMB2 Mean IO Operations/s (-5.4%) (-5.4%) Median (-3.7%) (-3.6%) 80 th (-3.1%) (-3.1%) 90 th (-3.0%) (-3.1%) 95 th (-3.1%) (-3.0%) 30

31 Percennt of Total # IOs OLTP More than 4/8 KiB Database File IO Size 10,000,000 1,000,000 8iKB 100,000 10,000 1, Read Write IO Size (KiBytes) Database File IO - Bytes Transferred by IO 100% 90% 80% 70% 60% Note: 49% 50% IO Size (KiBytes) Read Write 31

32 # IOs (Thousands) OLTP Log Matters 1000 Log Write Size IO Size (KiBytes) 32

33 OLTP Understanding workload is critical for performance Enabling new workloads is both a protocol and platform effort Windows 7 / 2008 R2 KB Mixed Read/Write Async NonCached IO Same File otherwise, no issue. It took a true database to hit it. No other workload affected!

34 Cluster Motion SMB 3.0 Continuous Availbility Cluster Implementations should look at performance of essential cluster management operations Bet: scale of clustered resources will rise significantly, relative to pre-smb 3.0 services Planned v. Unplanned Motion Goal: define the time budget.

35 Cluster Hardware Configuration CPU Type Memory Amount Network Type Number of Network Links Storage Adapter Storage Client Server 2 sockets with Ghz (4 cores total) 4GiB Onboard 1GbE network interface (Cluster Network) N/A N/A 1 x 1GbE 1 Fibre Channel Adapter 4Gb/s connectivity 120x 10KRPM HDD Enterprise FC Array Slow & Old! 35

36 OLTP Hardware Configuration 36

37 Cluster Motion Resource Group Container of objects representing the file service High scale on disks depending on implementation HW/SW Volume Smaller scale on total number of groups Config: 3 FS Groups, 40 disks each Net Name IPv4 IPv6 File Server Disk 2 Disk 1 Disk N 37

38 Cluster Motion Resource Control Manager handles the process for Windows, generalizable to other platforms Cluster log follows the state changes, similar to: PS> Get-ClusterLog RCM Offline Online WaitingToGoOffline OfflineCallIssued OfflinePending OfflineSavingCheckpoints RCM Online Offline WaitingToComeOnline OnlineCallIssued OnlinePending Online Offline b cc::2011/08/25-00:23: INFO [RCM] TransitionToState(IP Address ) OfflineCallIssued-->OfflineSavingCheckpoints.

39 Clock Time (seconds) Cluster Motion 60 Offline -> Online (Disks) - Initial

40 60000 Offline -> Online (NonDisk) Initial

41 Cluster Motion Crossover Ethernet cluster network Simultaneous disconnect of crossover network and loss of node (e.g., node power loss) Unplanned motion 80s delay Windows 2008 R2 KB

42 Clock Time (seconds) Cluster Motion - Disks 12 Hard Disk Offline - Initial Disks in 3 resource groups 42

43 Cluster Motion - Disks Hard disk resource internals Remove of FSCTL_LOCK_VOLUME prior to dismount Remove of timer used to simplify async completion of PR reservation release Remove of soft guard between IOCTL_OFFLINE_VOLUME and the online process Reduce conservative polling interval for disk onlines

44 Clock Time (seconds) Cluster Motion - Disks 12 Hard Disk Offline - Final Disks in 3 resource groups 44

45 Clock Time (seconds) Cluster Motion - Final 30 Offline -> Online (Disks) - Final Disks in 3 resource groups

46 Clock Time (seconds) Cluster Motion - Final 30 Offline -> Online (NonDisks) Final OnlinePending OnlineCallIssued WaitingToComeOnline Offline OfflineSavingCheckpoints OfflinePending OfflineCallIssued WaitingToGoOffline Online 46

47 Cluster Motion Summary Large outlier in unplanned loss removed Relative: ~80% faster at scale Absolute: sub-10s motion at scale Larger or smaller than possible reality? Implementers Look at your platform repeat under loads

48 Multichannel Goals Expand session beyond a single TCP stream / CPU Availability / Network fault tolerance Make the SMB2 protocol resilient to interface, link or switch failures. Move link awareness higher up the stack to enable more intelligent decision making. Augment NIC teaming at the network layer. Keep fallback paths ready, prioritize available links. React quickly to changes to network availability. Performance utilize the available resource 48

49 Multichannel - Terminology Channel: underlying transport connection Session: authenticated user context. Session Binding: map Session Channel N:N relationship User A User B Sessions Bindings TCP TCP RDMA Channels 49

50 Test Hardware Configuration CPU Type Memory Amount Network Type Client Server 2 sockets with Ghz (12 cores total) 48 GiB 2 network interface adapter cards Each card has 2 10GbE interfaces Number of Network Links Storage Adapter Storage N/A N/A 4 x 10GbE 2 RAID Host Bus Adapters 6Gb/s SAS connectivity 12 3G SATA SSD / HBA 50

51 Test Hardware Configuration RAID0 12 SSDs RAID0 12 SSDs Client 10GbE Server 51

52 IOMETER Configuration I/O Size Number of Workers Queue Depth Total Queued Constant #Op Constant Bytes Scaled Bytes 52

53 MB/sec Server (Local) Baseline Server/Local Read Throughput Read Sequential Read Random IO Size (bytes) 53

54 MB/sec SMB 3.0 Client 10GbE Interface Scaling 6000 SMB 3.0 Client Interface Scaling - Throughput 1 x 10GbE 2 x 10GbE 4 x 10GbE IO Size (bytes) 54

55 MB/sec Server to SMB 3.0 Client Comparison Server/Local vs. Client Throughput Server/Local Throughput Client (4 x 10GbE) Throughput IO Size (bytes) 55

56 I/Os per second Priv CPU % Server to SMB 3.0 Client IOps Server/Local vs. Client IOps with CPU % Server/Local IOps Client (4 x 10GbE) Priv CPU Client (4 x 10GbE) IOps Server/Local Priv CPU 350, , , , , ,000 50, IO Size

57 Multi Channel Implementation Challenges Nodes with varying interface counts (1, 2, 4 NIC) Operation distribution across interfaces TCP v. RDMA TCP CPU is in the host RDMA CPU is primarily in the HCA Different connection scale points Performance goal: scale

58 Summary SMB 3.0 enables new workloads Workload demands will expose new issues in platforms and implementations be prepared New protocol features create new performance requiremens Downlevel platform improvements Windows 2008 R2 KB Cluster Unplanned with Crossover Ethernet Windows 7 / 2008 R2 KB Single-File Mixed Read/Write Async NonCached IO