Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 SER2724BU Extreme Performance Series: Performance Best Practices Mark Achtemichuk, VCDX, Staff Engineer, VMware Reza Taheri, Principal Engineer, VMware Valentin Bondzio, Senior Staff TSE, VMware #VMworld #xperfseries #SER2724BU *

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #SER2724BU CONFIDENTIAL 2

3 Agenda 1 The Baseline 2 vnuma 3 Keeping Things Up To Date 4 Power Management 5 Hyper-threading 3

4 Baseline 4

5 Baseline Best Practices * Use the most current release: vsphere, VCSA, VM Tools, vhw, OS, BIOS, Firmware HW selection makes a difference ex: bandwidth, offloads, processor architectures Refer to existing best practice documentation ex: SQL BPs, Latency Sensitive BPs * Rightsize your workloads, size into a pnuma node, correct vcpu presentation * Evaluate your power management policy Use resource management properly, or not at all! * Keep Hyper-threading enabled Use DRS to manage contention 5

6 Baseline Best Practices Monitor oversubscription ex: pcpu:vcpu, memory reclamation, via vrops Use paravirtualized drivers: vmxnet3, pvscsi Evaluate disabling interrupt coalescing, lower latency, higher cost Storage design needs to be optimized for flash, map app -> disk Understand what the workload is java apps are different than databases Define and monitor application level KPIs You can t compare Apples to Oranges 6

7 Baseline Best Practices What s Important to Monitor: Compute Contention Ready, Co-Stop Memory Oversubscription Balloon, Swap (in-guest difficult) Storage Service Time Device and Kernel Latency Network Health - Throughput 7

8 Baseline Best Practices Performance Best Practices for vsphere Application Specific Best Practice Guides (SQL, Oracle, etc) VROOM! Blog Performance Community 8

9 Agenda 1 The Baseline 2 vnuma 3 Keeping Things Up To Date 4 Power Management 5 Hyper-threading 9

10 Troubleshooting Scenario vnuma Poor NUMA Locality (N%L) pnuma doesn t match vnuma I see conflicting guidance 10

11 Troubleshooting Scenario vnuma Optimal Configuration 11

12 Troubleshooting Scenario vnuma Rules of Thumb 1. While there are many advanced vnuma settings, only in rare cases do they need to be changed from defaults. 2. Always configure the virtual machine vcpu count to be reflected as Cores per Socket, until you exceed the physical core count of a single physical NUMA node. 3. When you need to configure more vcpus than there are physical cores in the NUMA node, evenly divide the vcpu count across the minimum number of NUMA nodes. 4. Don t assign an odd number of vcpus when the size of your virtual machine exceeds a physical NUMA node. 5. Don t enable vcpu Hot Add unless you re okay with vnuma being disabled* VMworld 2017 Content: Not for publication 6. Don t create a VM larger than the total number of physical cores of your host* 12

13 Agenda 1 The Baseline 2 vnuma 3 Keeping Things Up To Date 4 Power Management 5 Hyper-threading 13

14 Troubleshooting Scenario Why Keep Things Up-to-date In house developed application that is mission critical runs as expected on developer laptop but only about 70% of the performance on ESXi Analyze the problem Use tools to identify issue Lesson learned VMworld 2017 Content: Not for publication CONFIDENTIAL 14

15 Troubleshooting Scenario The developers laptop The ESXi server 15

16 Is There Contention? 10:05:56am up 2 days 43 min, 675 worlds, 1 VMs, 2 vcpus; CPU load average: 0.05, 0.01, 0.01 PCPU USED(%): AVG: 6.6 PCPU UTIL(%): AVG: 5.1 CORE UTIL(%): AVG: 11 ID GID NAME NWLD %USED %RUN %SYS %WAIT %VMWAIT %RDY %IDLE %OVRLP vmx NetWorld-VM NUMASchedRemapEpochInitial vmast vmx-vthread vmx-mks:prime vmx-svga:prime vmx-vcpu-0:prime vmx-vcpu-1:prime PVSCSI-96338: vmx-vthread-7:prime

17 Processor Differences? Both processors are Haswell, very similar clock frequencies Intel Core i7-4700mq (4 cores, 8 threads / 6 MB L3 Cache / 2.4 -> 3.4 GHz) Intel Xeon E v3 (6 cores, 12 threads / 15 MB L3 Cache / 2.4 -> 3.2 GHz) Is it EVC? 17

18 What About Virtual Hardware? ESXi vhw GA (ISO 8601) ~ Intel CPU gen. level Model Name mod (9) Ivy Bridge E*-**** v Haswell E*-**** v Skylake E*-**** v5 18

19 Virtual Hardware Version Changes Some 38 changes that were implemented in vhw 11 Examples: benefit SMT FT VMs (mostly SVGA optimizations) reduces timer interrupts for idle Windows VMs (less CPU consumption / contention when VMs are idle) enable RSC (LRO) for Windows VMs improves some aspect of nested VM performance etc. 19

20 Agenda 1 The Baseline 2 vnuma 3 Keeping Things Up To Date 4 Power Management 5 Hyper-threading 20

21 Troubleshooting Scenario Power Management Impact I run performance test on my software but always get varying results within runs. Could it be Power Management? Analyze the problem Use tools to identify issue Lesson learned VMworld 2017 Content: Not for publication CONFIDENTIAL 21

22 What is Power Management? Reallocating power consumption within a processor package More Turbo bins on cores in C0 when other cores are in deep C-States Frequency TB1 P0 TB1 P0 C1 C1 TB3 TB2 TB1 P0 TB3 TB2 TB1 P0 C6 C6 C-State depth C0 C1 C0 C6 22

23 Where Can I See Power Management? 10:15:23am up 2 days 53 min, 674 worlds, 1 VMs, 2 vcpus; CPU load average: 0.10, 0.09, 0.03 Power Usage: 147W, Power Cap: N/A PSTATE MHZ: CPU %USED %UTIL %C0 %C1 %C2 %P0 %P1 %P2 %P3 %P4 %P5 %P6 %P7 %P8 %P9 %P10 %P11 %P12 %P13 %A/MPERF ( ) ( ) 23

24 Balanced or High Performance? It Depends Always set BIOS to OS Controlled Then the policy change is dynamic Balanced (default) allows for Turbo opportunities Great for populations of small virtual machines Some performance variability is okay High Performance caps Turbo opportunities Best for populations that have Large VMs (greater than 8 vcpu) Required for Latency Sensitive workloads CONFIDENTIAL 24

25 Agenda 1 The Baseline 2 vnuma 3 Keeping Things Up To Date 4 Power Management 5 Hyper-threading 25

26 Troubleshooting Scenario Interfering VMs I ran my workload yesterday and it ran in 565 seconds; today it s 1096 seconds. What happened? Analyze the problem Use tools to identify issue Lesson learned VMworld 2017 Content: Not for publication CONFIDENTIAL 26

27 Problem definition The workload VM: 36 vcpus Running the bzip2 test of SPECcpu2006 Simple, no I/O But easy to see The Server: Broadwell-EP E GHz Turbo boost up to 2.8GHz for a 1.22X improvement 36 cores/72 HyperThreads Tools: Standard Linux tools inside the guest esxtop on the hypervisor Hardware event counters using PMC inside the guest Enable virtual Performance Monitoring Counters (vpmc), KB perf stat command to collect 27

28 Troubleshooting steps Fast case: 565 seconds mpstat(1) on the guest 100% CPU utilization All other guest tools report the same stats Slow case: 1096 seconds mpstat(1) on the guest 100% CPU utilization All other guest tools report the same stats Questions: Why did our performance drop by half? Can we reconcile guest stats with ESX stats? 28

29 Troubleshooting with esxtop In the fast case, we get the full utilization on all 36 cores The VM s %RUN is nearly 3600% Turbo boost of 22% (PCPU USED% to PCPU UTIL% ratio) matches the hardware counters In the slow case, we have a second VM running! Our %RUN is about half of available cycles, matching the hardware counters Our %READY and %COSTOP add up to about half the available cycles That s why the guest tools were fooled We need more CPUs!!! HyperThreading to the rescue!! One VM active Two VMs active 29

30 Intel Hyper-Threading Technology That doubles my CPUs, right? Not! VMworld 2017 Content: Not for Increases instruction level parallelism logic is replicated, partitioned, shared ~ 5% additional die size / cost publication ~ 25% more performance Most of the benefit comes from one HyperThread using the core while the other one is waiting for memory load Recent processors replicate some core functionality on each HyperThread 30

31 No problem: Enable HyperThreading! First, compare the one-vm case with and without HyperThreading Twice as many physical processors With HT, we have a new field: CORE UTIL% esxtop fields with HT enabled PCPU UTIL% versus CORE UTIL% with one HyperThread in use One HyperThread saturated keeps the core at 100% PCPU USED% reflects benefits of Turbo boost HT Disabled HT Enabled 31

32 What is PCPU USED % A measure of how much a given HyperThread uses the core A metric calculated by ESX When it s the only HyperThread active, it gets 100% credit Its twin gets 0% credit since it s idle During any periods that both HyperThreads are active, they each get 50% credit Execution time on the CPU One Hyperthread busy 100% PCPU USED Hyperthread 1 Hyperthreads 2 Execution time on the CPU Both Hyperthreads saturated 50% 50% PCPU USED PCPU USED Hyperthread 1 Hyperthreads 2 The average of PCPU USED credit of the two HyperThreads cannot go over 50% PCPU USED % is then adjusted up or down for Turbo and frequency scaling So with Turbo boost, the average of PCPU USED % of a core can go to, say, 2.8GHz/2.3GHz=61% Execution time on the CPU 100 Both Hyperthreads busy % PCPU USED 25% PCPU USED Hyperthread 1 Hyperthreads 2 32

33 HyperThreading with two VMs On this fully saturated Broadwell with HyperThreading enabled PCPU USED%: is 60% Discounted for HT; boosted for Turbo Our performance boost is only 1.16X despite CPU %RUN per VM has gone from ~1800% to ~3600% %READY and %COSTOP have largely disappeared The expected boost is 10-40% HT Disabled HT Enabled 33

34 Going a layer deeper Can we use other tools to diagnose why the HyperThreading boost is on the low side? Use Performance Monitoring Counters (PMC) built into the processor hardware PMCs can be virtualized on vsphere With vpmc, each VM only sees the event counts while it was running Analysis common for hard-core performance engineers CONFIDENTIAL 34

35 Enabling virtual Performance Monitoring Counters (vpmc) in a VM 35

36 Collecting hardware event counts in Linux # perf stat -a -e cycles -e ref-cycles -e instructions -e cache-misses -e cache-references sleep 10 Performance counter stats for 'system wide': 1,009,203,198,874 cycles 828,986,892,878 ref-cycles 1,456,882,482,627 instructions # 1.44 insn per cycle 984,566,743 cache-misses # % of all cache refs 14,516,610,550 cache-references seconds time elapsed Let s check the math for this workload that fully saturated all vcpu: 36 vcpus 2.3GHz 10 seconds = 828,000,000,000 36

37 Processor PMC stats with HyperThreading, one or two VMs HT VMs Overall thruput of all VMs PCPU UTIL % PCPU USED % CORE UTIL % Cycles/ second per VM Instr/ second per VM With HyperThreading Disabled, 1 VM and 2 VMs have nearly identical profiles With HyperThreading Enabled and 2 VMs, we get twice the cycles in VMs, but not the efficiency The L3 cache and Resource Stall stats tell us the VMs are interfering with each other IPC Total L3 cache accesses Total L3 cache misses Total Resource Stalls off % 121% - 2.7G 4.11G G 2.95G 442G off % 121% - 1.4G 2.15G G 3.01G 478G ON % 60% 99% 2.7G 2.44G G 7.25G 998G VMworld 2017 Content: Not for publication 37

38 Yes, that was complex! VMs interfere with each other I/O latency Cache hit rate Memory latency Core resources HyperThreading can give a boost Often X boost Have to use a variety of tools for a detailed analysis Looking just at mpstat or even CORE UTIL% and PCPU UTIL% would have been misleading Do not oversubscribe above available resources! 38

39 What we did NOT say We did not say don t overcommit Just that your VMs cannot use more resources than the hardware can offer We did not say don t believe Linux tools Just that when there is a discrepancy between mpstat and what the app claims, use esxtop to investigate We did not say use hardware event counters as a first resort It s a microscope PMC gives you insight into the processor, e.g. when VMs are interfering with other We said: performance troubleshooting can be complex. Use the tools available at different layers of vertical stack to get a full picture 39

40 Conclusion vsphere OOTB Performance is Excellent Tuning Required for Specific Workloads or Corner Cases Performance Requires Understanding HW to App Leverage Existing Best Practice Documentation for Support Performance is an Onion, Peel Back the Layers Links: #SER2724BU CONFIDENTIAL 40

41 Extreme Performance Series Las Vegas SER2724BU Performance Best Practices SER2723BU Benchmarking 101 SER2343BU vsphere Compute & Memory Schedulers SER1504BU vcenter Performance Deep Dive SER2734BU Byte Addressable Non-Volatile Memory in vsphere SER2849BU Predictive DRS Performance & Best Practices SER1494BU Encrypted vmotion Architecture, Performance, & Futures STO1515BU vsan Performance Troubleshooting VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks VIRT1397BU Optimize & Increase Performance Using VMware NSX VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX VIRT1052BU Monster VM Database Performance VIRT1983BU Cycle Stealing from the VDI Estate for Financial Modeling VIRT1997BU Machine Learning and Deep Learning on VMware vsphere FUT2020BU Wringing Max Perf from vsphere for Extremely Demanding Workloads FUT2761BU Sharing High Performance Interconnects across Multiple VMs #SER2724BU CONFIDENTIAL 41

42 Extreme Performance Series Barcelona SER2724BE Performance Best Practices SER2343BE vsphere Compute & Memory Schedulers SER1504BE vcenter Performance Deep Dive SER2849BE Predictive DRS Performance & Best Practices VIRT1445BE Fast Virtualized Hadoop and Spark on All-Flash Disks VMworld 2017 VIRT1397BE Optimize & Increase Performance Using VMware NSX VIRT1052BE Monster VM Database Performance FUT2020BE Wringing Max Perf from vsphere for Extremely Demanding Workloads Content: Not for publication #SER2724BU CONFIDENTIAL 42

43 Extreme Performance Series Hand on Labs Don t miss these popular Extreme Performance labs: HOL SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools. HOL CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. #SER2724BU CONFIDENTIAL 43

44 Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve. Scan this QR code to access a short survey and provide us direct feedback. Alternatively: Thank you! #SER2724BU CONFIDENTIAL 44

45

46