Understanding Your Virtual Workload. Irfan Ahmad CTO CloudPhysics

Transcription

1 What s PRESENTATION Your Shape? TITLE GOES 5 HERE Steps to Understanding Your Virtual Workload Irfan Ahmad CTO CloudPhysics

2 SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. 2

3 Abstract Is your Virtual Machine (VM) rightsized? Is your VM is on the right datastore? Is your VM IO bound? These are the kinds of questions that come up frequently when you re doing capacity management, solving performance problems, and making procurement decisions. The root of the answer to all of them is the shape of your workload. Learn you how to find the answers to workload shape questions. This tutorial delves into some of the challenges inherent in right-sizing virtual workloads by discovering the shape of your workload and applying that knowledge to capacity and performance decisions. Is the disk workload sequential or random? How much parallelism is there? How do we figure out the shape of a workload? What are the tools and techniques we can use? What is the bottleneck resource? How do we map workloads to the right mix of storage, CPU, memory and network? 3

4 The Virtualization Promise

5 No Longer Delivering 3

6 A New Set of Headaches How do I ensure that we meet our SLAs? Will I really benefit from costly SSD cache? Will performance suffer if I consolidate more? How do I properly plan my IT budget for next year?

7 Can t Datacenters Be? Designed Like Civil Engineering CAD software helps design infrastructure and model costs before building Operated Like Airlines Logistics management software allows for maximizing efficiency Predicted Like Chip Design Design automation software allows for testing before costly manufacturing Other large operations have powerful tools to design and manage, but datacenters do not.

8 Sure They Can! Example: Storage performance Predict how an SSD design will perform? Model the cost of operations and ROI? Tools now exist to design, predict and operate data centers. And workload shapes are the key!

9 Flash Overtakes In Mindshare Interest in flash memory has risen greatly, but it is costly and doesn t benefit everyone.

10 SSD has garnered a lot of hype The economics of flash memory are staggering. If you re not using SSD, you are doing it wrong. High Scalability Solid-state drive SSD Source: Google Trends, March 214 But does reality live up to the hype?

11 SSDs: A cautionary tale Company Quick Facts Light Vehicle Automotive Publicly traded Established 195s 4,+ employees $3bln+ revenue Large SSD caching project POC completed Production deployment completed But back-of-the-envelope VM selection VMs solely selected on application identity Project was a DISASTER VMs couldn t possibly benefit tremendous waster

12 Key questions Will SSDs benefit my datacenter? Which of my VMs / applications? How much cache do I need?

13 Do VMs benefit from SSDs? Depends Is the disk even a bottleneck (or is it CPU, memory)? How do you determine if a VM will benefit from caching? Detailed workload characterization Outstanding IOs analysis Read/write ratio analysis Latency analysis Cache hit ratio analysis No simple rule of thumb! No one size fits all

14 Workload Shapes Technique Histograms are much more informative than single numbers like mean, median, and standard deviations from the mean e.g., multimodal behaviors are easily identified by plotting a histogram, but obfuscated by a mean Histograms can actually be calculated efficiently online Why take one number if you can have a distribution? Mean is 5.3! 25 2 Made up Example Frequency Latency of an operation (microseconds) 8 9 1

15 Workload Shapes Technique The ESX disk IO workload characterization is on a pervirtual disk basis Allows us to separate out each different type of workload into its own container and observe trends Technique: For each virtual machine IO request in ESX, we insert some values into histograms E.g., size of IO request 4KB Data collected pervirtual disk

16 Workload Shapes Technique Read/Write Distributions are available for our histograms Overall Read/Write ratio? Are Writes smaller or larger than Reads in this workload? Are Reads more sequential than Writes? Which type of IO is incurring more latency? IO Size All, Reads, Writes Seek Distance All, Reads, Writes Seek Distance Shortest Among Last 16 Outstanding IOs All, Reads, Writes IO Interarrival Times All, Reads, Writes Latency All, Reads, Write

17 IO Length Filebench OLTP UFS Frequency I/O Length Histogram 4K and 8K IO transformed into 128K by ZFS? ZFS Frequency Length (bytes) I/O Length Histogram > > Length (bytes)

18 Seek Distance Filebench OLTP Seek Distance Histogram Seek distance: measure of sequentiality versus randomness in a workload Somehow a random workload is transformed into a sequential one by ZFS! More details needed... UFS ZFS Frequency Frequency Distance (sectors) Seek Distance Histogram Distance (sectors)

19 Seek Distance Filebench OLTP More Detailed Split out reads & writes 12 Seek Distance Histogram (Writes) 6 Seek Distance Histogram (Reads) 1 5 UFS Frequency Distance (sectors) Frequency Distance (sectors) Seek Distance Histogram (Writes) 3 Seek Distance Histogram (Reads) ZFS Frequency Frequency Distance (sectors) Distance (sectors) Transformation from Random to Sequential: primarily for Writes Reads: Seek distance is reduced (look at histogram shape & scales)

20 Filebench OLTP Summary So, what have we learnt about Filebench OLTP? IO is primarily 4K but 8K isn t uncommon (~3%) Access pattern is mostly random Reads are entirely random Writes do have a forward-leaning pattern ZFS is able to transform random Writes into sequential: Aggressive IO scheduling Copy-on-write (COW) technique (blocks on disk not modified in place) Changes to blocks from app writes are written to alternate locations Stream otherwise random data writes to a sequential pattern on disk Performed this detailed analysis in just a few minutes

21 OSDL Database Test 2 (Linux ) Analysis Workload is primarily random (big spikes towards the right and left edges of the graph) Still, many IOs that are within 5 sectors (2%) or within 5, sectors (33%) of the previous command The workload is almost exclusively 8K for both reads and writes Frequency Frequency Seek Distance Histogram (Writes) Distance (sectors) I/O Length Histogram > Length (bytes)

22 OSDL Database Test 2 (Linux ) Analysis (2) The number of outstanding IOs are very different in this workload between reads and writes PostgreSQL almost always issues 32 write IOs simultaneously IO rate from this workload varies over time as much as 15% over a 2 min period Frequency Outstanding I/Os Histogram (Reads, Writes) Outstanding I/Os Histogram over Time I/Os Outstanding at Arrival time Reads Writes 64 > 64 Frequency S16 S11 S6 S1 1 2 Time (in 6 sec intervals) > 64 I/Os Outstanding at Arrival time

23 OSDL Database Test 2 (Linux ) Summary On the aggregate the workload appears random But 2% of IOs are within 25KB and 33% are within 2.4MB! IO size is 8K for both reads and writes Outstanding IOs very different between reads and writes PostgreSQL almost always issues 32 write IOs simultaneously IO rate varies over time (up to 15%) Don t assume that every database workload behaves the same; measure and determine for yourself

24 Use Cases for Workload Shapes Analyzing a new disk performance sensitive workload Tuning of underlying disk subsystem How to interpret Pay attention to changes in distribution shape as well as magnitude Which metrics to start with IO Size Read/Write Ratios Outstanding IOs Corrective actions Tune disk subsystem and re-measure; pay attention to latency histogram

25 Workload shape matters Bursty writes Steady read traffic 8K reads and writes Bimodal spatial locality The read/write ratio is highly biased towards reads. Understanding application IO patterns is the first step in predicting SSD benefits.

26 Limitation of Simple Shape Analysis Strength: allows deep analysis of separate flavors of workloads in each VM by splitting workloads by virtual disk Place DB redo logs on a separate virtual disk than the DB tablespaces Weakness: doesnt give a complete picture of IO going to a storage array Many VMs might be doing IO from same ESX host VMs from different ESX hosts might be doing IO In general, it is a hard problem to figure out Rule of thumb: IO to a LUN from different apps is effectively random Still: storage arrays are rather smart to pull off individual sequential streams and schedule IO per stream

27 Disk Access Traces Matter Even More (Source: USENIX 6) Knowing patterns isn t enough. Exact IO sequences are required.

28 Algorithms to the rescue Hit Ratio Curves Simulation Prediction Algorithms Big gains at ~5MB and 22MB, but little in between. Data access patterns, IO sequences and complex analytics allow for maximizing ROI of SSD cache.

29 SSDs: A success story Company Quick Facts Boston-area hedge fund International operations Assets >$2B Established 198s 5+ employees 1s of VMs analyzed 16% showed benefit from server-side SSD cache Improvement ranged from 5% to 2% better response times Hit Ratio Curve derived cache size recommendations ranged from 1GB - 512GB (VM-by-VM basis) Successfully identified VMs that benefit from SSD cache, SSDs in 16% of VMs.

30 More success: SSDs in 3% of VMs COMPANY QUICK FACTS Public University (Boston area) Established 185s 1, students 13+ employees 1s of virtual machines simulated 3% of VMs had 5% or higher improvement Hit Ratio Curves derived recommendations ranged from 1GB 512GB (VM-by-VM basis) Customer installed 2 PCIe Flash cards to get maximum benefit via a strategic installation Even smaller deployments benefit.

31 PRESENTATION APPENDIX TITLE GOES HERE 31

32 Attribution & Feedback The SNIA Education Committee thanks the following individuals for their contributions to this Tutorial. Authorship History Additional Contributors Name/Date of Original Author here: Irfan Ahmad, CloudPhysics, April 17, 214 Please send any questions or comments regarding this SNIA Tutorial to 32

33 Workload Characterization Technique To make the histograms practical, bin sizes are on rather irregular scales E.g., the IO length histogram bin ranges like this:, 248, 495, 496, 8191, 8192, rather odd: some buckets are big and others are as small as just 1 Certain block sizes are really special since the underlying storage subsystems may optimize for them; single those out from the start (else lose that precise information) E.g., important to know if the IO was 16KB or some other size in the interval (8KB,16KB)

34 Windows File Copy XP issues 64KB IOs IOs are largely sequential. Frequency Seek Distance Histogram Vista Enterprise XP Pro Vista is issuing very large IOs (1MB) Distance (bytes) Number of commands is lower IOs are very sequential Latency is higher Vista enables large IOs to be issued; file copy is just an example Frequency I/O Length Histogram Length (bytes) Vista Enterprise XP Pro > Keep an eye out for increasing IO sizes in future workloads

35 How to Run $ /usr/lib/vmware/bin/vscsistats -p iolength Histogram: IO lengths of commands { min : 512 max : mean: count : 241 { 5 (<= 512) 14 (<= 124) 5 (<= 248) 17 (<= 495) 76 (<= 496) 1 (<= 8191) 2 (<= 8192) 18 (<= 16383) 36 (<= 16384) 49 (<= 32768) (<= 49152) (<= 65535) (<= 65536) (<= 8192) (<= 13172) (<= ) (<= ) (> ) } } $ /usr/lib/vmware/bin/vscsistats -p latency Histogram: latency of IOs in Microseconds (us) { min : 191 max : mean: 598 count : 288 { (<= 1) (<= 1) (<= 1) 248 (<= 5) 28 (<= 1) 4 (<= 5) 8 (<= 15) (<= 3) (<= 5) (<= 1) (> 1) } } Bin Ranges (Bucket Limits). Think x-axis of histograms plots

36 Performance Overhead of Stats Collection Overhead is negligible (tested on internal build) Online Histo Service Disabled Enabled IOps IOps Std. Dev MBps CPU (out of 8) CPU Std. Dev CPU Efficiency (UsedSec/IOps) Latency (ms) Table 2. Microbenchmark Performance Used iometer to generate 4KB Sequential Reads 16 outstanding IOs on a Windows 23 Enterprise Edition 64-bit VM 4KB is the most realistic worst-case scenario for overheads