Reducing Memory Virtualization Overheads in Virtualized Datacenters. Jayneel Gandhi, Arkaprava Basu, Michael M. Swift, Mark D.

Size: px

Start display at page:

Download "Reducing Memory Virtualization Overheads in Virtualized Datacenters. Jayneel Gandhi, Arkaprava Basu, Michael M. Swift, Mark D."

Ariel Price
5 years ago
Views:

1 Reducing Memory Virtualization Overheads in Virtualized Datacenters Jayneel Gandhi, Arkaprava Basu, Michael M. Swift, Mark D. Hill

2 Executive Summary Problem: TLB misses in virtual machines Hardware-virtualized MMU has high overheads Up to 280% overhead Prior Work: Direct Segments unvirtualized case Solution: segmentation to bypass paging Extend Direct Segments for virtualization Three configurations with different tradeoffs Results Near- or better-than-native performance 2

3 Outline Motivation Review: Direct Segments Virtualized Direct Segments Optimizations Evaluation Methodology Results Summary 3

4 Overheads of Virtualizing Memory We show that that the increase in translation lookaside buffer (TLB) miss-handling costs due to hard-ware-assisted memory management unit (MMU) is the largest contributor to the performance gap between native and virtual servers. Buell, et al. VMware Technical Journal

5 Unvirtualized x86 translation Virtual Address VA Page Table CR3 Physical Address PA Up to mem accesses = 4 5

6 Two-levels of translation Guest Virtual Address Guest Physical 1 2 Address Host Physical Address cr3 cr3 gva gpa hpa Guest Page Table Nested Page Table Base Virtualized 6

7 Support for Virtualizing Memory gva CR3 hpa Up to Mem accesses gpa 5 ncr3 gpa ncr3 + 5 gpa ncr3 + 5 gpa ncr3 + 5 gpa ncr3 + 4 = 24 7

8 Applications Database Graph-analytics Key-value store HPC Apps Big-memory applications Standard Workloads (in-memory apps) 8

9 EXECUTION TIME OVERHEAD EXECUTION TIME OVERHEAD Cost of Virtualization 60% 50% 40% Native Virtual Overheads of virtual memory can be high 800% 700% 600% 500% 30% 400% 20% 10% 300% 200% 100% 0% 0% graph500 memcached NPB: CG GUPS 9

10 EXECUTION TIME OVERHEAD EXECUTION TIME OVERHEAD 113% 1556% Cost of Virtualization Native Virtual 60% 50% Overheads of virtualizing virtual memory are even higher 800% 700% 600% 40% 500% 30% 400% 20% 300% 200% 10% 100% 0% 0% graph500 memcached NPB: CG GUPS 10

11 EXECUTION TIME OVERHEAD EXECUTION TIME OVERHEAD 113% 1556% 880% 908% Cost of Virtualization Native Virtual 60% 50% Larger pages at VMM are not sufficient to achieve native performance 800% 700% 600% 40% 500% 30% 400% 20% 300% 200% 10% 100% 0% 0% graph500 memcached NPB: CG GUPS 11

12 EXECUTION TIME OVERHEAD EXECUTION TIME OVERHEAD 113% 1556% 880% 908% Cost of Virtualization Native Virtual 60% 50% Large Pages (B) are not sufficient to achieve negligible overheads 800% 700% 600% 40% 500% 30% 400% 20% 300% 200% 10% 100% 0% 0% graph500 memcached NPB: CG GUPS 12

13 EXECUTION TIME OVERHEAD EXECUTION TIME OVERHEAD 113% 1556% 880% 908% Cost of Virtualization Native Virtual 60% 50% Huge Pages (B) can adversely affect performance 800% 700% 600% 40% 500% 30% 400% 20% 300% 200% 10% 100% 0% 0% graph500 memcached NPB: CG GUPS 13

14 THP THP+ THP+ THP THP+ THP+ THP THP+ THP+ THP THP+ THP+ THP THP+ THP+ THP THP+ THP+ 0.59% 0.81% 1.14% EXECUTION TIME OVERHEAD 83% 88% 103% 70% 280% 149% 160% Cost of Virtualization 60% 50% Native Virtual Similar trends for standard workloads 40% 30% 20% 10% 0% cactusadm canneal GemsFDTD mcf omnetpp streamcluster 14

15 Outline Motivation Review: Direct Segments Virtualized Direct Segments Optimizations Evaluation Methodology Results Summary 15

16 Unvirtualized Direct Segments 1 Conventional Paging 2 Direct Segment VA BASE OFFSET LIMIT PA Why Direct Segment? Matches big memory workload needs NO TLB lookups => NO TLB Misses Basu et al. [ISCA 2013] 16

17 Translation with Direct Segments [V 47 V 46 V 13 V 12 ] [V 11 V 0 ] OFFSET BASE? LIMIT<? Direct Segment Ignored N HIT HIT/MISS DTLB Lookup MISS Page-Table Walker [P 40 P 39.P 13 P 12 ] Basu et al. [ISCA 2013] [P 11 P 0 ] 17

18 Translation with Direct Segments [V 47 V 46 V 13 V 12 ] [V 11 V 0 ] OFFSET BASE? Y LIMIT<? DTLB Lookup Paging Ignored HIT/MISS MISS Page-Table Walker [P 40 P 39.P 13 P 12 ] Basu et al. [ISCA 2013] [P 11 P 0 ] 18

19 Direct Segments VA VA cr3 cr3 PA Base Native PA Direct Segments 19

20 Outline Motivation Review: Direct Segments Virtualized Direct Segments Evaluation Methodology Results Summary 20

21 Guest Page Table Modes Nested Page Table 1 2 gva cr3 gpa cr3 hpa Base Guest VMM Dual Virtualized Direct 21

22 Base Virtualized: Translation gva cr3 1 gpt gpa cr3 2 npt hpa Base Virtualized 22

23 Dual Direct: Translation gva cr3 1 gpt Guest Direct Segment gpa cr3 2 npt Host Direct Segment hpa Dual Direct 23

24 VMM Direct: Translation gva cr3 1 gpt gpa cr3 2 npt Host Direct Segment hpa VMM Direct 24

25 Guest Direct: Translation gva cr3 1 Guest Direct Segment gpa cr3 2 npt hpa 25

26 Tradeoffs: Efficiency Properties Dimension of page walk # of mem. accesses for most page walks # of base-bound computation for page walks Base Virtualized Dual Direct VMM Direct Guest Direct 2D 0D 1D 1D

27 Tradeoffs: Compatibility Properties Guest OS modifications VMM modifications Application Category Base Dual VMM Guest Virtualized Direct Direct Direct X X X minimal Any Any Bigmemory Bigmemory 27

28 Tradeoffs: Memory Overcommit Properties Base Virtualized Dual Direct VMM Direct Guest Direct Page Sharing limited limited Ballooning limited limited Guest OS Swapping limited limited VMM Swapping limited limited 28

29 Outline Motivation Review: Direct Segments Virtualized Direct Segments Optimizations Evaluation Methodology Results Summary 29

30 Optimizations Issue 1: Guest/Host memory fragmentation Guest physical memory: Self-ballooning Host physical memory: Compaction Issue 2: Permanent hard memory faults Escape filter to provide alternate translation Bloom filter stores small number of faulty pages Filter checked in parallel with segment registers For more details: Come to the poster 30

31 Outline Motivation Review: Direct Segments Virtualized Direct Segments Optimizations Evaluation Methodology Results Summary 31

32 Methodology Estimate performance of future hardware Measure fraction of TLB misses to segmented memory Measure TLB miss cost with performance counters Estimate performance gain with linear model Prototype Linux v host/guest Qemu-KVM hypervisor Intel 12-core Sandy-bridge with 96GB memory 32

33 +VD GD+ DD +VD GD+ DD +DS GD+ DD +VD GD+ DD EXECUTION TIME OVERHEAD EXECUTION TIME OVERHEAD 113% 1556% 880% 908% Results Native Virtual Modeled 60% 800% 50% 700% 600% 40% 500% 30% 400% 20% 300% 200% 10% 100% 0% 0% graph500 memcached NPB: CG GUPS 33

34 +VD GD+ DD +VD GD+ DD +DS GD+ DD +VD GD+ DD EXECUTION TIME OVERHEAD EXECUTION TIME OVERHEAD 113% 1556% 880% 908% Results Native Virtual Modeled 60% 50% VMM Direct and Guest Direct achieves near-native performance 800% 700% 600% 40% 500% 30% 400% 20% 300% 200% 10% 100% 0% 0% graph500 memcached NPB: CG GUPS 34

35 +VD GD+ DD +VD GD+ DD +DS GD+ DD +VD GD+ DD 0.01% 0.00% 0.01% 0.17% EXECUTION TIME OVERHEAD EXECUTION TIME OVERHEAD 113% 1556% 880% 908% Results Native Virtual Modeled 60% 50% Dual Direct eliminates most of the TLB misses achieving better-than native performance 800% 700% 600% 40% 500% 30% 400% 20% 300% 10% 200% 100% 0% 0% graph500 memcached NPB: CG GUPS 35

36 +VD +VD +VD +VD +VD +VD EXECUTION CYCLE OVERHEAD 83% 88% 103% 60% 70% 280% 149% 160% 82% Results 60% 50% Native Virtual Modeled VMM Direct achieves near-native performance for standard workloads 40% 30% 20% 10% 0% cactusadm canneal GemsFDTD mcf omnetpp streamcluster 36

37 Summary Problem: TLB misses in virtual machines Hardware-virtualized MMU has high overheads Up to 280% overhead Prior Work: Direct Segments unvirtualized case Solution: segmentation to bypass paging Extend Direct Segments for virtualization Three configurations with different tradeoffs Results Near- or better-than-native performance 37

38 Questions? For more details: Come to the poster 38

Virtual Virtual Memory

Virtual Virtual Memory Jason Power 3/20/2015 With contributions from Jayneel Gandhi and Lena Olson 4/17/2015 UNIVERSITY OF WISCONSIN 1 Virtual Machine History 1970 s: VMMs 1997: Disco 1999: VMWare (binary