Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 STO1515BU Extreme Performance Series: vsan Performance Troubleshooting Amitabha Banerjee and Suraj Kasi #VMworld #STO1515BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. 2

3 Agenda 1 vsan Day 1 Performance 2 Debugging and Improving vsan Performance 3

4 vsan 6.6 Is Loaded with Performance vsan 6.6 delivers 50%+ performance improvements Enhancements behind performance improvements Optimizations in caching checksums Optimizations in Write buffer de-staging Improvements in Dedup & Compression IO path Improvements in Object placement 4

5 vsan 6.6 Performance Improvements vs vsan 6.5 IOPS Improvement % Erasure Coding + Dedup Improvements in: High VM consolidation Transactional workloads Batch jobs on large files/ datasets Erasure Coding Default Features Checksum Disabled Random Read Random Write Mixed 70Read/30Write Data collected using FIO Benchmark on a 4-node vsan cluster, 2 dgs per node, 60% capacity usage 5

6 vsan Day 1 Performance

7 So You Just Deployed vsan How fast can it go? What is the best IOPs? How about throughput and latency? 7

8 But, Benchmarking is Non-trivial Which benchmark? HCIBench/ Hammer DB/ FIO/ TPCE etc. Am I running the benchmark the right way? Have I configured vsan the right way? Am I interpreting the results correctly? Is this the best that vsan can do? 8

9 Introducing vsan Performance Diagnostics vsan 6.6.1, vsphere 6.5 U1 Evaluate Benchmarks for a specific Goal: IOPS, Throughput, and Latency Select a Time Range during which benchmark ran for analysis Detect Performance Issues using vsan Performance Diagnostics Ask VMware documents possible recommendations Hopefully, recommendations help you: Fine tune benchmark parameters Configure vsan optimally Alter SPBM policy parameters to achieve desired goal 9

10 vsan Performance Diagnostics Salient Features Cloud Service Enable CEIP + vcenter Internet connectivity ( No sensitive information collected ) VMware can ship updates independent of on-prem installation. Integrated with VMware internal performance debugging Fully integrated with vsan Performance Service Enable vsan Performance Service Explore more data at a mouse click 90 days of data. Explore 1 day at a time Integrated with HCIBench HCIBench Time Ranges automatically populated Consume results within HCIBench 10

11 vsan Performance Diagnostics Demo Run HCIBench with Random Write Workload Analyze with vsan Performance Diagnostics Tweak vsan / Workload Rerun, Re-iterate 11

12 12

13 13

14 Improvement in HCIBench Result IOPS % Latency IOPS 95 percentile Latency (ms) VM, 1 disk, 2 threads, Stripes=1, FTT=1 1 VM, 1 disk, 2 threads, Stripes=7, FTT=1 0 14

15 15

16 Improvement in HCIBench Result IOPS % 1600% Latency IOPS 95 Percentile Latency (ms) 0 1 VM, 1 disk, 2 threads, Stripes=1, FTT=1 1 VM, 1 disk, 2 threads, Stripes=7, FTT=1 8 VM, 1 disk, 8 threads, Stripes=7, FTT=1 0 16

17 Debugging and Improving vsan Performance VMworld 2017 Content: Not for publication

18 VSAN Architecture Overview DOM Comp. Mgr LSOM VSCSI DOM / vsan Client DOM Owner DOM Comp. Mgr LSOM PSA PSA VSCSI: Virtual SCSI DOM: Distributed Object Manager Client: Provide access to VSCSI/other clients Owner: Arbitrate access to objects, enforce policy Component Manager: Handles vsan components LSOM: Log Structured Object Manager Handles objects in cache and capacity PSA/Device Layer 18

19 Debugging and Improving vsan Performance Resync Throttling Hardware Configuration Unaligned IO Congestion 19

20 Example: Resync Throttling Typical resync workflows One or more node or disk failures Node or disk evacuation VM Storage policy reconfiguration Cluster rebalancing & disk format upgrades Designed for shortest resync time Resync traffic can impact VM IO Use the throttling mechanism to control the resync speed For, vsan6.6 or vsphere65ep2 End user control over resync activity through UI For, Pre vsan6.6 vsi config option called MaxinflightIO If resync is slow, reduce the VM IO traffic 20

21 21

22 Debugging and Improving vsan Performance Resync Throttling Hardware Configuration Unaligned IO Congestion 22

23 vsan Observer Capture & visualize vsan performance statistics. Collected in Ruby vsphere Console (RVC) $rvc Or $(windows)%programfiles%\vmware\infrastructure\virtualcenter Server\support\rvc\rvc.bat $vsan.observer -r -k./vmworldtestbed/ usage: observer [opts] cluster -r, --run-webserver Run a webserver to view live stats -k, --keep-observation-in-memory or f output_filename --max-diskspace-gb=<i>, --interval=<i>, --generate-html-bundle=<s> Access the live data via web URL: 23

24 Hardware Configuration Sizing Undersized vsan might not get the performance you expect Choose the right CPU, Memory, Cache, Capacity and Disk Group sizes Driver Firmware Compatibility Check vsan Health Check VMware Compatibility Guide for vsan(vcg) Incompliance can result in huge performance implications 24

25 Example: NVMe Driver Certain Intel NVMe drives don t perform the best with inbox NVMe driver Large sequential write workload will cause performance issue For releases vsphere6.5 and before, Intel NVMe driver (not inbox driver) is recommended 25

26 Wrong NVMe Driver DOM NVMe vsanowner Client Disks Latency 20ms 60ms vsan Disk IOPS 5,000 Read latency 10ms Write latency 4ms 26

27 Correct NVMe Driver NVMe vsan Disks latency 3ms vsan Disks IOPS 20,000 Read latency 1.2ms Write latency 0.2ms 27

28 Debugging and Improving vsan Performance Resync Throttling Hardware Configuration Unaligned IO Congestion 28

29 Know Your Workload IOInsight What does your IO workload look like? Understand your application better with IOInsight IOInsight monitors IO pattern and provides high-level insights on your application Metrics IO size distribution Read/Write ratio Sequentiality 4k-alignment Data fill rate Cache friendliness Download fling: 29

30 IOInsight Can be used as pre-deployment tool to correctly configure vsan Read/Write Ratio: Write-intensive applications can benefit from using >1 disk group Read / Write intensive workload benefit from higher stripe width 4k-Alignment Expect reduced performance if IOs are misaligned Data fill rate Roughly estimate rate of space growth for capacity planning Cache friendliness Estimate space requirement for caching tier for hybrid vsan setups 30

31 Example: Unaligned IO 4k block size was introduced in vsan disk format version 3.0 Unaligned IO causes IO Inflation VMDK vsan Physical Disk 4k Block 4k Write k Block VMDK 1 Write vsan IO Inflation. 2 Reads Modify 2 Writes 31

32 Unaligned IO vsan DOM CPU Owner Disks Utilization Latency Low 0.5ms 20ms Network Errors - Low 32

33 Unaligned IO IO Insight 33

34 Unaligned IO IO Insight Write Intensive workload 34

35 Unaligned IO IO Insight 4KB Write Workload 35

36 Unaligned IO IO Insight 4KB Sequential Write Workload 36

37 Unaligned IO IO Insight 4KB Unaligned Sequential Write 37

38 Debugging and Improving vsan Performance Resync Throttling Hardware Configuration Unaligned IO Congestion 38

39 Congestion Flow control mechanism used by vsan Bottleneck in lower layer is relieved through controlling the rate of incoming IO at the vsan ingress Types of congestion SSD congestion Cache tier write buffer space runs out Comp congestion IO activity on a particular vsan component is exceeding the threshold Mem congestion Memory heap usage by vsan internal components exceeds the threshold IOPS congestion IOPS on vsan components exceeds reservations Log congestion Occurs when vsan internal log space usage in cache tier disk runs out or Encourage you attend vsan Beyond the Basics [STO1479BU] talk distribution 39

40 Example: Log Congestion Occurs when vsan internal log space usage in cache tier disk runs out Observed with write-intensive workloads and deduplication & compression are turned on Log compaction Efficient utilization of the available log space Introduced in vsphere 6.0 Update 3 and vsan 6.6 (vsphere 6.5 EP2) Software update will unleash better performance 40

41 Log Congestion Log congestion

42 Conclusion vsan 6.6 delivers 50% bettter performance vsan Performance Diagnostics available in vsan Integrated with HCIBench vsan Performance Debugging Tools vsan Performance Service Graphs vsan Observer vsan Health Checks IO Insight 42

43 Extreme Performance Series Las Vegas SER2724BU Performance Best Practices SER2723BU Benchmarking 101 SER2343BU vsphere Compute & Memory Schedulers SER1504BU vcenter Performance Deep Dive SER2734BU Byte Addressable Non-Volatile Memory in vsphere SER2849BU Predictive DRS Performance & Best Practices SER1494BU Encrypted vmotion Architecture, Performance, & Futures STO1515BU vsan Performance Troubleshooting VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks VIRT1397BU Optimize & Increase Performance Using VMware NSX VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX VIRT1052BU Monster VM Database Performance VIRT1983BU Cycle Stealing from the VDI Estate for Financial Modeling VIRT1997BU Machine Learning and Deep Learning on VMware vsphere FUT2020BU Wringing Max Perf from vsphere for Extremely Demanding Workloads FUT2761BU Sharing High Performance Interconnects across Multiple VMs 43

44 Extreme Performance Series Barcelona SER2724BE Performance Best Practices SER2343BE vsphere Compute & Memory Schedulers SER1504BE vcenter Performance Deep Dive SER2849BE Predictive DRS Performance & Best Practices VIRT1445BE Fast Virtualized Hadoop and Spark on All-Flash Disks VMworld 2017 VIRT1397BE Optimize & Increase Performance Using VMware NSX VIRT1052BE Monster VM Database Performance FUT2020BE Wringing Max Perf from vsphere for Extremely Demanding Workloads Content: Not for publication 44

45 Extreme Performance Series Hand on Labs Don t miss these popular Extreme Performance labs: HOL SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools HOL CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems 45

46 Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve. Scan this QR code to access a short survey and provide us direct feedback. Alternatively: Thank you! VMworld 2017 Content: Not for publication 46

47

48