vcacheshare: Automated Server Flash Cache Space Management in a Virtualiza;on Environment

Transcription

1 vcacheshare: Automated Server Flash Cache Space Management in a Virtualiza;on Environment Fei Meng Li Zhou Xiaosong Ma Sandeep U3amchandani Deng Liu With VMware during this work

2 Background Virtualization Apps Apps Apps Guest OS Hypervisor USENIX ATC'14 2

3 Background Virtualization Server Flash Cache Apps Apps Apps Guest OS (Server Flash Cache) SFC Hypervisor USENIX ATC'14 2

4 Background Virtualization Server Flash Cache I/O acceleranon Apps Apps Apps Guest OS Increased VM- to- server consolidanon ranos (Server Flash Cache) SFC Hypervisor Reduced disk I/O load Reduced inter- server I/O contennon USENIX ATC'14 2

5 Problem Setting and Motivations SFC management VM1 VM2 VM3 VM4 VM5 Hypervisor SFC USENIX ATC'14 3

6 Problem Setting and Motivations SFC management Static space partitioning VM1 VM2 VM3 VM4 VM5 SFC1 SFC2 SFC3 SFC4 SFC5 Hypervisor SFC USENIX ATC'14 3

7 Problem Setting and Motivations SFC management Static space partitioning Globally shared VM1 VM2 VM3 VM4 Shared Hypervisor VM5 SFC USENIX ATC'14 3

8 Problem Setting and Motivations SFC management Static space partitioning Globally shared Issues Different access behavior across VMs Temporal changes in locality VM1 VM2 VM3 VM4 Shared Hypervisor VM5 SFC USENIX ATC'14 3

9 Problem Setting and Motivations SFC management Static space partitioning Globally shared Issues Different access behavior across VMs vcacheshare Temporal changes in locality VM1 VM2 VM3 VM4 SFC1 SFC2 SFC3 SFC4 SFC5 Hypervisor VM5 SFC USENIX ATC'14 3

10 vcacheshare Design Goal Automated and continuously optimized SFC space allocation, considering USENIX ATC'14 4

11 vcacheshare Design Goal Automated and continuously optimized SFC space allocation, considering VM priority Locality of workloads IO access characteristics Backend device service time Configuration events (e.g., migration) USENIX ATC'14 4

12 vcacheshare (vcs) Overview Dynamic cache partitioning based on Continuous per-vm access history collection and analysis Comprehensive cache utility model How useful is more cache space to this workload? Combining hit ratio and reuse intensity in cache utility calculation Long-term vs. short-term observation Different turnaround requirement for access history analysis USENIX ATC'14 5

13 vcacheshare Design Cluster User world Cluster management node Administration events such as VMotion VM VMVM Cache utility analyzer RD array CUs Database vcacheshare Optimizer linear, nonlinear solver etc I/O I/O logs RD Hypervisor VMKernel Cache module Cache management (bookkeeping, hash, LRU list management etc.) Intercept I/O log Monitor trace buffers Resize actions Recommended cache allocation plan Actuator I/O I/O Hardware SSD vmdk vmdk SAN, NAS USENIX ATC'14 6

14 vcacheshare Design Cluster User world Cluster management node Administration events such as VMotion VM VMVM Cache utility analyzer RD array CUs Database vcacheshare Optimizer linear, nonlinear solver etc ESXi Hypervisor VMKernel Cache module I/O Cache management (bookkeeping, hash, LRU list management etc.) I/O logs RD Intercept I/O log Monitor trace buffers Resize actions Recommended cache allocation plan Actuator I/O I/O Hardware SSD vmdk vmdk SAN, NAS USENIX ATC'14 6

15 vcacheshare Design Cluster User world Cluster management node Administration events such as VMotion VM VMVM Cache utility analyzer RD array CUs Database vcacheshare Optimizer linear, nonlinear solver etc ESXi Hypervisor VMKernel Cache module I/O Cache management (bookkeeping, hash, LRU list management etc.) I/O logs RD Intercept I/O log Monitor trace buffers Resize actions Recommended cache allocation plan Actuator I/O I/O Hardware SSD vmdk vmdk SAN, NAS USENIX ATC'14 6

16 vcacheshare Design Cluster User world Cluster management node Administration events such as VMotion VM VMVM Cache utility analyzer RD array CUs Database vcacheshare Optimizer linear, nonlinear solver etc ESXi Hypervisor VMKernel Cache module I/O Cache management (bookkeeping, hash, LRU list management etc.) I/O logs RD Intercept I/O log Monitor trace buffers Resize actions Recommended cache allocation plan Actuator I/O I/O Hardware SSD vmdk vmdk SAN, NAS USENIX ATC'14 6

17 vcacheshare Design Cluster User world Cluster management node Administration events such as VMotion VM VMVM Cache utility analyzer RD array CUs Database vcacheshare Optimizer linear, nonlinear solver etc ESXi Hypervisor VMKernel Cache module I/O Cache management (bookkeeping, hash, LRU list management etc.) I/O logs RD Intercept I/O log Monitor trace buffers Resize actions Recommended cache allocation plan Actuator I/O I/O Hardware SSD vmdk vmdk SAN, NAS USENIX ATC'14 6

18 vcacheshare Design Cluster User world Cluster management node Administration events such as VMotion VM VMVM Cache utility analyzer RD array CUs Database vcacheshare Optimizer linear, nonlinear solver etc ESXi Hypervisor VMKernel Cache module I/O Cache management (bookkeeping, hash, LRU list management etc.) I/O logs RD Intercept I/O log Monitor trace buffers Resize actions Recommended cache allocation plan Actuator I/O I/O Hardware SSD vmdk vmdk SAN, NAS USENIX ATC'14 6

19 vcacheshare Design Cluster User world Cluster management node Administration events such as VMotion VM VMVM Cache utility analyzer RD array CUs Database vcacheshare Optimizer linear, nonlinear solver etc ESXi Hypervisor VMKernel Cache module I/O Cache management (bookkeeping, hash, LRU list management etc.) I/O logs RD Intercept I/O log Monitor trace buffers Resize actions Recommended cache allocation plan Actuator I/O I/O Hardware SSD vmdk vmdk SAN, NAS USENIX ATC'14 6

20 vcs Component 1: Cache Module or l Cache module Cache management (bookkeeping, hash, LRU list management etc.) I Cache module USENIX ATC'14 7

21 vcs Component 1: Cache Module or l Cache module Cache management (bookkeeping, hash, LRU list management etc.) I Write-around read cache VM read Cache module SFC read write Disk array Cache policy illustranon USENIX ATC'14 7

22 vcs Component 1: Cache Module or l Cache module Cache management (bookkeeping, hash, LRU list management etc.) I Write-around read cache VM read Cache module Block-level cache targeting hypervisor SFC write read Disk array Cache policy illustranon USENIX ATC'14 7

23 vcs Component 1: Cache Module or l Cache module Cache management (bookkeeping, hash, LRU list management etc.) I Write-around read cache VM read Cache module Block-level cache targeting hypervisor Dynamically resizable SFC write read Disk array Cache policy illustranon USENIX ATC'14 7

24 vcs Component 1: Cache Module or l Cache module Cache management (bookkeeping, hash, LRU list management etc.) I Write-around read cache VM read Cache module Block-level cache targeting hypervisor Dynamically resizable VM-aware SFC write read Disk array Cache policy illustranon USENIX ATC'14 7

25 vcs Component 1: Cache Module or l Cache module Cache management (bookkeeping, hash, LRU list management etc.) I Write-around read cache VM read Cache module Block-level cache targeting hypervisor Dynamically resizable VM-aware LRU SFC read Disk array write Cache policy illustranon USENIX ATC'14 7

26 vcs Component 2: Monitor Monitor trace buffers vcs monitor USENIX ATC'14 8

27 vcs Component 2: Monitor Monitor trace buffers vcs monitor Location: In I/O path, between VMs and SFC USENIX ATC'14 8

28 vcs Component 2: Monitor Monitor trace buffers vcs monitor struct cache_io_stats { unit16 VM_UUID; unit16 VMDK_UUID; unit16 Nmestamp; uint8 isread; uint32 LBA; uint8 len; uint16 latency; uint8 iscached; } Cache stat entry Location: In I/O path, between VMs and SFC Intercepts I/O accesses for trace collection In-memory plus on-ssd circular buffer Copy segments (sliding window) to Analyzer s buffers Time window *Analyzer *SSD buffer Copy to In- memory buffer USENIX ATC'14 8

29 vcs Component 2: Monitor Monitor trace buffers vcs monitor struct cache_io_stats { unit16 VM_UUID; unit16 VMDK_UUID; unit16 Nmestamp; uint8 isread; uint32 LBA; uint8 len; uint16 latency; uint8 iscached; } Cache stat entry Location: In I/O path, between VMs and SFC Intercepts I/O accesses for trace collection In-memory plus on-ssd circular buffer Copy segments (sliding window) to Analyzer s buffers Overhead tolerable for workloads tested Sampling possible if necessary Time window *Analyzer *SSD buffer Copy to In- memory buffer USENIX ATC'14 8

30 vcs Component 3: Analyzer Processes trace segments from Monitor Output USENIX ATC'14 9

31 vcs Component 3: Analyzer Processes trace segments from Monitor Output Reuse distance CDF Used in calculating Hit Ratio (HR) Measuring long-term locality behavior Commonly used in cache partitioning [Mattson1970][Ding2003] But incomplete: does not capture access speed USENIX ATC'14 9

32 vcs Component 3: Analyzer Processes trace segments from Monitor Output Reuse distance CDF Used in calculating Hit Ratio (HR) Measuring long-term locality behavior Commonly used in cache partitioning [Mattson1970][Ding2003] But incomplete: does not capture access speed 2 hours VM1: VM2: seconds OpNmizaNon Nme USENIX ATC'14 9

33 vcs Component 3: Analyzer Processes trace segments from Monitor Output Reuse distance CDF Used in calculating Hit Ratio (HR) Measuring long-term locality behavior Commonly used in cache partitioning [Mattson1970][Ding2003] But incomplete: does not capture access speed Reuse Intensity (RI) Measuring how fast are data re-visited Estimated by ratio of total accessed volume and footprint S total /( tw Sunique) Coarse and more frequent measurement USENIX ATC'14 9

34 vcs Component 4: Optimizer For each VM Cache utility Backend device latency EsNmated hit rano CU i (c) = l i RR i (HR i (c)+αri i ) Reuse intensity Read rano Tuning knob USENIX ATC'14 10

35 vcs Component 4: Optimizer For each VM Cache utility Optimization Objective function max n i=1 CU i (c) = l i RR i (HR i (c)+αri i ) priority i CU i (c) Constraints c + c +! + c = 1 2 n C c c USENIX ATC'14 10 min i c max Backend device latency Read rano EsNmated hit rano Tuning knob Reuse intensity

36 vcs Component 4: Optimizer For each VM Cache utility Optimization Objective function max n i=1 CU i (c) = l i RR i (HR i (c)+αri i ) priority i CU i (c) Constraints c + c +! + c = 1 2 n C c c min i c max Backend device latency Read rano Simulated annealing EsNmated hit rano Tuning knob Reuse intensity <c 1, c 2, c 3, > Adjusted cache size USENIX ATC'14 10

37 vcacheshare ESX Event Handling VM Events VM Power- Off, Suspend, vmonon Source VM Bootstrap, Power- On, Resume vmonon DesNnaNon AcNons by vcacheshare Framework Trigger opnmizanon to re- allocate; free IO trace buffers for all associated VMDKs Make ininal allocanon based on reserved cache space and priority seings; start trace collecnon Trigger opnmizanon to re- allocate based on IO characterisncs migrated with the VMDKs involved in vmonon Storage vmonon (runnme backend de- vice change) VM Fast Suspend, vmonon Stun Suspend analysis/opnmizanon Nll complenon; evict device service latency history; trigger re- opnmizanon upon vmonon complenon Reserve cached data; lock cache space allocanon to involved VMs by subtracnng allocated size from total available cache size USENIX ATC'14 11

38 Evaluation Setup Prototype VMware EXSi 5.0 with ~2800 LOC C++ user space code and ~2500 LOC kernel C code Hardware Two AMD Opteron 8-core CPU, 16GiB memory, Intel 400GiB SSD EMC Clariion array VM configuration 1vCPU, 1GiB memory and 8/80GiB VMDK Win2k8, Ubuntu USENIX ATC'14 12

39 Micro-Benchmark: Iometer Cache allocation (%) VM1 footprint shrunk VM2 replay VM1 GLRU VM2 GLRU VM1 vcs VM2 vcs Time (s) USENIX ATC'14 13

40 Micro-Benchmark: Iometer Cache allocation (%) VM1 GLRU VM2 GLRU VM1 vcs VM2 vcs Cache allocanon Time (s) VM1 GLRU VM2 GLRU VM1 vcs VM2 vcs IO latency (ms) Time (s) Workload latency USENIX ATC'14 13

41 Latency (ms) Macro-benchmark: VDI 100 VDI VMs going through boot storm + 1 Iometer VM GLRU vcs Static No cache StaNc: 72s Time (s) vcs: 114s GLRU:242s No cache:298s USENIX ATC'14 14

42 Macro-benchmark: VDI 100 VDI VMs going through boot storm + 1 Iometer VM Latency (ms) GLRU vcs Static No cache VDI latency Latency (ms) Time (s) Time (s) vcs VDI Static VDI Iometer latency USENIX ATC'14 14

43 Closing Remarks Dynamic SFC partitioning affordable and worthwhile Long sampling windows desirable for accurate cache hit ratio estimation But cause slow response to locality spikes Can be compensated by simultaneous short-term locality monitoring Future work: relationship between sampling window size and cache hit ratio estimation accuracy USENIX ATC'14 15

44 Q&A Fei Meng North Carolina State University USENIX ATC'14 16

45 References [Mattson1970] R. L. Mattson, J. Gecsei, D. R. Slutz, and I. L. Traiger. Evaluation techniques for storage hierarchies. IBM Syst. J., 9(2):78 117, June [Ding2003] C. Ding and Y. Zhong. Predicting Wholeprogram Locality through Reuse Distance Analysis. In ACM SIGPLAN Notices USENIX ATC'14 17