Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 STO2591BU Amplifying VMware vsan Performance: A Deep Dive into Testing and Tuning (with Lessons Learned) by Western Digital Jonathan Flynn Virtualization Technologist, Western Digital Chen Wei Sr. Solution Architect, VMware #VMWORLD #STO2591BU #WesternDigiDC

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. 2

3 Resources: Blogs Cormac Hogan: Director and Chief Technologist in Office of CTO of the Storage & Availability BU at VMware Duncan Epping: Chief Technologist in the Office of CTO of the Storage & Availability BU at VMware Virtual Blocks: Jonathan Flynn: Virtualization Technologist in the Data Propulsion Labs Western Digital Chen Wei: Senior Solutions Architect VMware 3

4 Resources: Publications Essential Virtual SAN Second Edition: VMware vsan Design and Sizing Guide 6.5: VMware vsan Network Design: VMware Virtual SAN 6. Performance: 4

5 Agenda 1 Introduction to Western Digital vsan Performance Testbed 2 HCIBench 3 Using HCIBench to Find Your Clusters Sweet Spot 4 Western Digital vsan Performance Testbed Configuration 5 vsan Performance Knobs 6 vsan Features 7 vsan Networking 5

6 Introduction to Western Digital vsan Performance Testbed VMworld 217 Content: Not for publication

7 The Challenge How can Western Digital better support the vsan community? Partners developing VMware vsan ReadyNodes Direct or Channel Customers building their own vsan with certified components How does Western Digital s product lineup fit into vsan designs? With many new products being released in 217, Western Digital needed a clear picture of where those new devices fit. Ultrastar SS2, Ultrastar SS3, Ultrastar SN2, Ultrastar SN26, Skyhawk, Ultrastar He12 How can Western Digital gain and provide a better understanding of the following? vsan Performance Tuning vsan Feature Performance and Usability vsan System and Networking Design

8 The Solution Providing a standardized testing environment and automating performance testing allows: Device performance analysis between products and product generations The ability to do feature analysis, design principle testing, and performance tuning characterization which then allows: Feedback and guidance on product selection to partners and customers at large Internal engineering feedback (for Western Digital use) A greater understanding of the vsan SDS stack to share through blogs VMworld 217 Content: Not for publication 8

9 Testing Methodology HCIBench Initial workload configured and automated Simple to automate and provides excellent ability to test environment to ensure it is performing as expected DiskSpd in Windows Guest Future workload to implement Provides alternative synthetic performance to HCIBench s vdbench/linux combo VMMark 3. Future workload to implement Provides deeper insight into application performance More difficult to implement, as it is a full test suite using multiple benchmarks tools Microsoft SQL Server HammerDB Future workload to implement Will provide TPC-C like and TPC-H like performance results showing mission-critical applications on vsan

10 HCIBench 1

11 Introduction to HCIBench Developed by VMware specifically for vsan performance testing Can be used with other HCI environments on top of VMware Essentially an automation wrapper around the popular and proven Vdbench open source benchmark tool HCIBench fully automates the end-to-end testing process: Deploying test VMs across the cluster Launching and coordinating workload runs (vsan) Collecting vsan Observer raw data during test runs Aggregating test results Collecting necessary data for troubleshooting purposes Composed of VMware s PhotonOS and Opensource VDbench

12 HCIBench vsan Specific Integration Uses vsan Observer for detailed data collection during runs Can clear/flush vsan read and write caches between runs to ensure consistent, accurate results Integrates with vsan Performance Diagnostic for the latest release. Provides appropriate fill processes to ensure all VMDKs have been fully provisioned before testing begins Includes appropriate fill for deduplication and compression testing or distribution

13 HCIBench Architecture Guest vm Guest vm vsphere Guest vm Guest vm Guest vm vsan or Other Datastore Testing Results RVC Guest vm Template VMworld 217 Content: Not for Vdbench Workload File publication Automation Bundle Config File Web UI HCIBench Appliance 13

14 HCIBench Workflow Only for easy run to test against vsan Environment Identification Client VMs Deployment VMDK Blocks Allocation If Needed Clear Read/Write Cache/Buffer Available when testing against vsan I/O Testing Results Collection and Performance Diagnostic 14

15 Using HCIBench to Find Your Cluster s Sweet Spot VMworld 217 Content: Not for publication 15

16 What is a Sweet Spot in this context? Latency How OIO impact performance Outstanding IO IO Per Second Latency IOPS

17 What is a Sweet Spot in this context? Percentage Change Normalized as Percentage Change Outstanding IO IOPS Latency

18 Sweet Spot process Decide your workload Run Benchmark VMworld 217 Analyze Result (Sweet Spot Found Here) Modify Workload # of VM # of VMDK Content: Not for publication # of Threads 18

19 Workload Definition for HCIBench and vsan Defined as Outstanding I/Os (OIO) per host or Total Effective Queue Depth per host For HCIBench this is a function (Number of VMs * Number of Data Disks per VM * Threads per Data Disk) / Number of Hosts This is a combination of HCIBench configuration (VMs and Data Disks) and VDBench configuration files (Threads per Data Disk) OIO relationship to block size: As your block size increases, your OIO should consider decreasing. Example: A sweet spot of 512 OIO for a 4KB block size will not work for a 1MB block size. VMworld 217 Content: Not for publication 19

20 Identifying Western Digital vsan Performance Testbed Sweet Spot For the Western Digital vsan Performance Testbed we targeted the OIOs below for the initial run. We then doubled and quadrupled these OIO numbers to determine the sweet spot. Remember your cluster may require reducing the workload or halving and quartering. Workload Size Target OIO per Host 2x Target 4x Target 4KB KB KB-32KB KB KB and larger Note: Due to having 2 VMs per host and 8 VMDKs per VM, the minimum OIO per configurable host is 16. Winner 2

21 Identifying Western Digital vsan Performance Testbed Sweet Spot 1% Read IOPS and Latency vsan 6.5 IOPs or distribution K 32K 64K 256K 512K 1M Block Size Target 2x Target 4x Target Target - Lat 2x Target - Lat 4x Target - Lat

22 Identifying Western Digital vsan Performance Testbed Sweet Spot 1% Write IOPS and Latency vsan 6.5 IOPs or distribution K 32K 64K 256K 512K 1M Block Size Target 2x Target 4x Target Target - Lat 2x Target - Lat 4x Target - Lat

23 Western Digital vsan Performance Testbed Configuration VMworld 217 Content: Not for publication

24 Hardware Configuration All-Flash vsan (4) Node Lenovo X365 M5 (2) E V3 14-core 2.6GHz (8) 32GB for 256GB total ConnectX -3 Lenovo MLOM (2) 56 Gb/s Ethernet connections (1..2) Avago 93-8i USB boot drive vsphere 6.5 Update 1 VSAN disk configuration (2) Disk groups per node (1) 1.92TB * Ultrastar SS2 cache drive (2) 7.68TB Ultrastar SS2 capacity drive * 1TB=1,GB, 1GB=1,,, bytes. Actual usable capacity less.

25 HCIBench Configuration Checked Clear Read/Write Cache Before Each Testing Checked Re-Use The Existing VMs If Possible Set - Initialize Storage Before Testing Zero all or Random for deduplication and compression testbed Vdbench Guest VM specification VMDK size calculation: (8DG * 6GB *.75Ratio)/(FTT + 1)/NumVM/NumDataDisk 36GB/2/8/8 = = 29GB per Data Disk 8 29GB (1.2) PVSCSI Controllers 2 VMs per host = 8 VMs

26 vsan Performance Knobs

27 vsan Performance Knobs Working set size This ties to the application you ll be running on vsan, or to your active dataset size. If your dataset(s) fit within cache (even in all-flash) you ll maintain a higher performance. Additional disk groups vsan software design lends itself to providing better performance with additional disk groups. All-flash example: (4) disk groups of (1) 3GB cache drive and (2) capacity drives will generally perform better than (2) disk groups of (1) 6GB Cache drive and (4) Capacity drives, even though the same number of capacity drives are used, and the caching capacity is the same in both configurations.

28 vsan Performance Knobs VMworld 217 Stripe width Relates to adding a stripe to a mirror or parity set. Increasing the strip width with fault tolerance set to RAID 1 will result in a RAID 1. Increasing the strip width with fault tolerance set to RAID 5 or 6 will result in a RAID 5 or 6. This is done via storage policies: Content: Not for publication

29 HCIBench Results Performance Knobs 1% Read IOPs and Latency IOPs vsan MS 4K 8K 32K 64K Block Size Baseline 18TB Baseline Stripe Width 4 3 Disk Groups 4 Disk Groups Baseline - Lat 18TB Baseline - Lat Stripe Width 4 - Lat 3 Disk Groups - Lat 4 Disk Groups - Lat

30 HCIBench Results Performance Knobs 1% Write IOPs and Latency IOPs vsan MS 4K 8K 32K 64K Block Size Baseline 18TB Baseline Stripe Width 4 3 Disk Groups 4 Disk Groups Baseline - Lat 18TB Baseline - Lat Stripe Width 4 - Lat 3 Disk Groups - Lat 4 Disk Groups - Lat

31 vsan - Features

32 vsan - Features Tests Covered Checksum Deduplication & Compression Erasure Coding RAID5 and RAID6 *Backup slides provide examples of how to enable/disable each feature with links to Cormac s blog covering benefits of each feature 32

33 HCIBench Results - Features 1% Read IOPs and Latency IOPs %-15% Increase With Checksum Disabled ONLY RECOMMEND IN LAB ENVIRONMENT VMworld 217 Content: Not for 15%-68% Increase With Dedup and Comp Disabled vsan publication K 8K 32K 64K Block Size Baseline Checksum Dedup & Compression Baseline - Lat Checksum - Lat Dedup & Compression - Lat

34 HCIBench Results - Features 1% Write IOPs and Latency IOPs %-32% Increase With Checksum Disabled VMworld 217 Content: Not for vsan publication 6%-11% Increase With Dedup and Comp Enabled K 8K 32K 64K Block Size Baseline Checksum Dedup & Compression Baseline - Lat Checksum - Lat Dedup & Compression - Lat

35 HCIBench Results - Features Erasure Coding 1% Read IOPs and Latency IOPs K 8K 32K 64K Block Size 4-Node Baseline 8-Node Baseline Raid 5 Erasure Coding Raid 6 Erasure Coding 4-Node Baseline - Lat 8-Node Baseline - Lat Raid 5 Erasure Coding - Lat Raid 6 Erasure Coding - Lat vsan MS

36 HCIBench Results - Features Erasure Coding 1% Write IOPs and Latency IOPs K 8K 32K 64K Block Size 4-Node Baseline 8-Node Baseline Raid 5 Erasure Coding Raid 6 Erasure Coding 4-Node Baseline - Lat 8-Node Baseline - Lat Raid 5 Erasure Coding - Lat Raid 6 Erasure Coding - Lat vsan MS

37 vsan Network

38 vsan - Network Tested: 9 MTU vs 15MTU 1Gb Ethernet vs 56Gb Switch Flow Control Dual vsan VMKernel Adapters Both 56Gb and 1Gb

39 vsan - Network Dual vsan VMKernel Adapters Port Group Virtual Switch Port Group These are not necessarily for redundancy (like an Air-Gap network with redundant physical interfaces routed to multiple VMKs) but for performance to pull from two physical interfaces at once. I found I was not coming close to the 56Gb network interfaces I had deployed. So, what would happen if I deployed two VMKernel adapters?

40 HCIBench Results Network 1% Read IOPs and Latency IOPs K 8K 32K 64K Block Size vsan Baseline Multiple vsan VMK 15 MTU 1Gb Ethernet 1Gb Eth Multiple vsan VMK Baseline - Lat Multiple vsan VMK - Lat 15 MTU - Lat 1Gb Ethernet - Lat 1Gb Eth Multiple vsan VMK - Lat MS

41 HCIBench Results Network Design Implications 64K 1% Read Network Activity Baseline - 17 Gb/s Receive and 17.8 Gb/s Transmit 4 Disk Group 17.3 Gb/s Receive and 25 Gb/s Transmit

42 HCIBench Results Network 1% Write IOPs and Latency IOPs K 8K 32K 64K Block Size vsan Baseline Multiple vsan VMK 15 MTU 1Gb Ethernet 1Gb Eth Multiple vsan VMK Baseline - Lat Multiple vsan VMK - Lat 15 MTU - Lat 1Gb Ethernet - Lat 1Gb Eth Multiple vsan VMK - Lat MS

43 HCIBench Results Network Design Implications 64K 1% Write Network Activity - Baseline Baseline - 1 Gb/s Receive and 8.7 Gb/s Transmit 4 Disk Group 16.9 Gb/s Receive and 15.3 Gb/s Transmit

44 Takeaways CPU: vsan can be highly dependent on core-frequency. I recommend fewer cores at higher frequency. Memory Example: Choose Intel Xeon Gold Core 3.GHz over the Xeon Platinum Core 2.1GHz Plan for vsan memory footprint. Memory footprint is directly impacted by the number of disk groups, caching device capacity, and all-flash cache settings, among other design and configuration choices. During the 4 disk group testing we found that vsan would only allow us to create 3 disk groups and then fail on the 4th. This was vsan 6.5 with caching devices of 1.75TB and the cache settings set for all-flash. This turned out to be a max memory overhead setting. /LSOM/maxSlabMemory Default 6.5: 2 Default 6.6: 3 To increase: esxcli system settings advanced set -o "/LSOM/maxSlabMemory" --int-value "64" 44

45 Takeaways Network 1Gb Ethernet is definitely a minimum requirement for an all-flash vsan. 25Gb Ethernet is my recommendation for an all-flash vsan, as it is easy with SSDs today to saturate the 1Gb links. Jumbo Frames is desirable. Reduced CPU overhead and increased performance makes it a very attractive feature. Disk Groups One of the easiest ways to boost performance on vsan, provided the systems and networks are wellthought-out, is to add additional disk groups per node. You wont see linear scalability, but you should see performance improvement For all-flash this includes breaking larger disk groups into smaller disk groups if you can, keeping in mind memory overhead and the maximum of 5 disk groups per node. Example: 2 disk groups of 1 caching device and 4 capacity devices would translate into 4 disk groups of 1 caching device and 2 capacity devices. As with everything in this world, there are exceptions 45

46

47 Booth #525 VMworld 217 Content: Not for publication

48 HCIBench 6.5 vs % Read IOPs IOPs K 8K 32K 64K Block Size Baseline Cache Settings Baseline Raid 5 Erasure Coding Raid 5 Erasure Coding Raid 6 Erasure Coding Raid 6 Erasure Coding Checksum Checksum Dedup & Compression Dedup & Compression

49 HCIBench 6.5 vs % Write IOPs IOPs K 8K 32K 64K Block Size Baseline Cache Settings Baseline Raid 5 Erasure Coding Raid 5 Erasure Coding Raid 6 Erasure Coding Raid 6 Erasure Coding Checksum Checksum Dedup & Compression Dedup & Compression

50 Infrastructure Configuration Network: MTU 9 Virtual Distributed Switch Dual 56Gb Ethernet uplinks vmk Management/VM traffic uplink 1 preference VLAN 152 DHCP lab network vmk1 vsan traffic uplink 2 preference VLAN 3 Private Static vmk2 vmotion Uplink 1 Preference VLAN 4 Private Static Send & receive flow control enabled at switch VMworld 217 Content: Not for Second cluster (multicast configuration): esxcli vsan network ipv4 set -d u i vmk1 VSAN Modify VSAN default storage policy to disable checksum DR (dedupe and compression) disabled Automatic Disk Claim disabled publication ESXi post-install NTP client active Power policy set to High Performance IPv6 disabled SSH and TSM enabled Required for HCIBench

51 vsan - Network Jumbo Frames 9 MTU These can be difficult to deploy end-to-end in a large-scale network. Understanding Ethernet frames and header sizes is required to properly set switches. Reduces CPU overhead Requires configuration at: Physical switch port VMKernel adapter Virtual switch VMworld 217 Content: Not for publication

52 vsan - Network 1Gb Network & Network Flow control 1Gb Network With the furtherance of SDS solutions and the removal of storage bottlenecks, appropriate network selection is becoming more and more important. All-flash vsan requires 1Gb. How does 1Gb compare with the 56Gb I have configured. Network Flow Control VMware recommends implementing flow control based on the vsan Networking Guide. My test environment did not prove the flow control to be a benefit, but this would be the case in an isolated environment where there are not other demands on the network.

53 vsan Features Checksum

54 vsan Features RAID 5 / 6 Erasure Coding FTT = 2 for RAID6

55 vsan Features De-duplication and Compression

56 Design (4) Grantley-EP dual-socket 1U or 2U servers, each with: (2) E V3 or E5-2697A V4 (8) 32GB DDR or DDR4-24 or 256GB memory SD card or USB boot device Dual-port Mellanox ConnectX -3 Pro 4Gb/56Gb Ethernet adapter (2) Broadcom LSI 93-8I for SAS and SATA configurations (2) Disk groups per node: generic, all-flash configuration (1) Caching tier device (2) Capacity tier devices 56

57 Identifying Western Digital vsan Performance Testbed Sweet Spot 1% Read Bandwidth MB/s vsan 6.5 4K 32K 64K 256K 512K 1M Block Size Target 2x Target 4x Target

58 Identifying Western Digital vsan Performance Testbed Sweet Spot 1% Write Bandwidth MB/s vsan 6.5 4K 32K 64K 256K 512K 1M Block Size Target 2x Target 4x Target

59 MB/s Identifying Western Digital vsan Performance Testbed Sweet Spot Mixed Workload Bandwidth Target 2x Target 4x Target IOPS IOPS and Latency vsan MS Target 2x Target 4x Target Target - Lat 2x Target - Lat 4x Target - Lat 2 8k 7% Read Workload 64k 95% Read 1 2 Workload

60 HCIBench Results Performance Knobs 1% Read Bandwidth MB/s vsan K 8K 32K 64K Block Size Baseline Max Cache Settings 6.5 Max Cache Settings TB Baseline Stripe Width 4 3 Disk Groups 4 Disk Groups

61 HCIBench Results Performance Knobs 1% Write Bandwidth MB/s vsan K 8K 32K 64K Block Size Baseline Max Cache Settings 6.5 Max Cache Settings TB Baseline Stripe Width 4 3 Disk Groups 4 Disk Groups

62 MB/s HCIBench Results Performance Knobs Mixed Workload Bandwidth Baseline Max Cache Settings 6.5 Max Cache Settings TB Baseline Stripe Width 4 3 Disk Groups 4 Disk Groups IOPs IOPs and Latency MS vsan Baseline Max Cache Settings 6.5 Max Cache Settings TB Baseline Stripe Width 4 3 Disk Groups 4 Disk Groups Baseline - Lat Max Cache Settings Lat Max Cache Settings Lat 18TB Baseline - Lat Stripe Width 4 - Lat 3 Disk Groups - Lat 4 Disk Groups - Lat 8k 7% Read Workload 64k 95% Read 8k 7% Read 64k 95% Read Workload

63 HCIBench Results - Features 1% Read Bandwidth MB/s vsan K 8K 32K 64K Block Size Baseline Checksum Dedup & Compression

64 HCIBench Results - Features 1% Write Bandwidth MB/s vsan K 8K 32K 64K Block Size Baseline Checksum Dedup & Compression

65 MB/s HCIBench Results - Features Mixed Workload Bandwidth Baseline Checksum Dedup & Compression IOPs IOPs and Latency vsan MS Baseline Checksum Dedup & Compression Baseline - Lat Checksum - Lat Dedup & Compression - Lat k 7% Read Workload 64k 95% Read 1 2 Workload

66 HCIBench Results - Features Erasure Coding 1% Read Bandwidth MB/s vsan K 8K 32K 64K Block Size 4-Node Baseline 8-Node Baseline Raid 5 Erasure Coding Raid 6 Erasure Coding

67 HCIBench Results - Features Erasure Coding 1% Write Bandwidth MB/s vsan K 8K 32K 64K Block Size 4-Node Baseline 8-Node Baseline Raid 5 Erasure Coding Raid 6 Erasure Coding

68 MB/s HCIBench Results - Features Erasure Coding Mixed Workload Bandwidth 4-Node Baseline 8-Node Baseline Raid 5 Erasure Coding Raid 6 Erasure Coding IOPs IOPs and Latency MS vsan Node Baseline 8-Node Baseline Raid 5 Erasure Coding Raid 6 Erasure Coding 4-Node Baseline - Lat 8-Node Baseline - Lat Raid 5 Erasure Coding - Lat Raid 6 Erasure Coding - Lat 8k 7% Read Workload 64k 95% Read 1 2 Workload

69 HCIBench Results Network 1% Read Bandwidth MB/s vsan K 8K 32K 64K Block Size Baseline Multiple vsan VMK 15 MTU 1Gb Ethernet 1Gb Eth Multiple vsan VMK

70 HCIBench Results Network 1% Write Bandwidth MB/s vsan K 8K 32K 64K Block Size Baseline Multiple vsan VMK 15 MTU 1Gb Ethernet 1Gb Eth Multiple vsan VMK

71 MB/s HCIBench Results Network Mixed Workload Bandwidth Baseline Multiple vsan VMK 15 MTU 1Gb Ethernet 1Gb Eth Multiple vsan VMK IOPs IOPs and Latency MS vsan Baseline Multiple vsan VMK 15 MTU 1Gb Ethernet 1Gb Eth Multiple vsan VMK Baseline - Lat Multiple vsan VMK - Lat 15 MTU - Lat 1Gb Ethernet - Lat 1Gb Eth Multiple vsan VMK - Lat k 7% Read 64k 95% Read Workload 8k 7% Read 64k 95% Read Workload