Virtual Virtual Memory

Size: px

Start display at page:

Download "Virtual Virtual Memory"

Ginger Tate
5 years ago
Views:

1 Virtual Virtual Memory Jason Power 3/20/2015 With contributions from Jayneel Gandhi and Lena Olson 4/17/2015 UNIVERSITY OF WISCONSIN 1

2 Virtual Machine History 1970 s: VMMs 1997: Disco 1999: VMWare (binary translation) 2003: Xen (para-virtualization) 2006(ish): Hardware support 4/17/2015 UNIVERSITY OF WISCONSIN 2

3 1974 VM Origins 4/17/2015 UNIVERSITY OF WISCONSIN 3

4 Virtual Machine Monitor (VMM) P1 P2 P1 P2 OS 1 OS 2 CPU Virtual Mem Machine Monitor I/O dev Disk VMM also called a hypervisor 4/17/2015 UNIVERSITY OF WISCONSIN 4

5 Disco Trap and emulate P1 P2 P1 P2 OS 1 OS 2 Virtual Machine Monitor CPU Mem I/O dev Disk 4/17/2015 UNIVERSITY OF WISCONSIN 5

6 What about x86? x86 can t use trap and emulate Classic Example: popf instruction Same instruction behaves differently depending on execution mode User Mode: changes ALU flags Kernel Mode: changes ALU and system flags Does not generate a trap in user mode 4/17/2015 UNIVERSITY OF WISCONSIN 6

7 VMWare Solution: binary translation Only need to translate OS code Makes SPEC run fast by default Most instruction sequences don t change Instructions that do change: Indirect control flow: call/ret, jmp PC-relative addressing Privileged instructions 4/17/2015 UNIVERSITY OF WISCONSIN 7

8 Overheads Traps are heavy weight Binary translation Bad for OS-heavy workloads (many server apps) What if you re allowed to change OS a little? 4/17/2015 UNIVERSITY OF WISCONSIN 8

9 Paravirtualization and Xen Use hypercalls to bypass VMM Still emulate for corner cases & safety reasons P1 OS 1 P2 P1 OS 2 P2 Commonly used! Amazon EC2 Virtual Machine Monitor CPU Mem Not full virtualization I/O dev Disk 4/17/2015 UNIVERSITY OF WISCONSIN 9

10 Another ring Hardware Support int moves from user-mode to kernel-mode vmrun moves from kernel-mode to vmm-mode Many other instructions What about virtual memory? 4/17/2015 UNIVERSITY OF WISCONSIN 10

11 Let s recall virtual memory P1 P OS 1 0 4/17/2015 UNIVERSITY OF WISCONSIN 11

12 Virtualize the OS s memory? OS 1 OS Hypervisor 0 4/17/2015 UNIVERSITY OF WISCONSIN 12

13 Two dimensions of translation P1 P2 P1 P2 OS 1 Virtual addresses Guest physical addresses OS Hypervisor Physical addresses 0 4/17/2015 UNIVERSITY OF WISCONSIN 13

14 Two dimensions of translation Guest Virtual Address Guest Physical 1 Address 2 Host Physical Address gva cr3 gpa cr3 hpa Guest Page Table Nested Page Table 14

15 What about the TLB? Want to cache virtual machine in TLB (Relatively) Easy with software-loaded TLBs TLB miss is a trap (virtual guest physical) Guest OS loads TLB (VMM trap) Translates guest physical machine physical VMM actually does the TLB insert Problem? 4/17/2015 UNIVERSITY OF WISCONSIN 15

16 Hardware walked pagetable Page table walker walks nested pagetable Need a fake page table 4/17/2015 UNIVERSITY OF WISCONSIN 16

17 Shadow Paging Guest Page Table 1 2 Nested Page Table gva cr3 gpa cr3 hpa VMM creates shadow page tables cr3 VMM keeps them coherent Shadow Page Table Keeping shadow page table coherent introduces overheads 17

18 Hardware support Today s hardware is aware of nested pagetable Nested page table walk For each level, must do a full pagetable walk Can be very high overhead 4/17/2015 UNIVERSITY OF WISCONSIN 18

19 Support for Virtualizing Memory gva CR3 hpa gpa ncr3 gpa ncr3 gpa ncr3 gpa ncr3 gpa ncr3

20 Tradeoffs Nested Paging Up to 24 memory references Shadow Paging Up to 4 memory references Updates to either page tables without VMM intervention Updates to either page tables requires costly VMM intervention Beneficial with Low TLB miss rate High page table updates Beneficial with High TLB miss rate Low page table updates 20

21 Cost of virtualization Overheads (native) Execution time overhead 100% 80% 60% 40% 20% 0% 4K 4K+4K 4K+2M 2M 2M+2M 4K 4K+4K 4K+2M 2M 2M+2M 4K 4K+4K 4K+2M 2M 2M+2M 4K 4K+4K 4K+2M 2M 2M+2M 1200% 1000% 800% 600% 400% 200% 0% graph500 memcached NPB:CG gups 4/17/2015 UNIVERSITY OF WISCONSIN 21

22 Cost of virtualization Execution time overhead 100% 80% 60% 40% 20% 0% 202% 4K 4K+4K 4K+2M 2M 2M+2M Overheads (native) 4K 4K+4K 4K+2M 2M 2M+2M Overheads (virtualized) 4K 4K+4K 4K+2M 2M 2M+2M 4K 4K+4K 4K+2M 2M 2M+2M 1200% 1000% 800% 600% 400% 200% 0% Execution time overhead graph500 memcached NPB:CG gups 4/17/2015 UNIVERSITY OF WISCONSIN 22

23 Cost of virtualization Execution time overhead 100% 80% 60% 40% 20% 0% 202% 126% 4K 4K+4K 4K+2M 2M 2M+2M Overheads (native) 4K 4K+4K 4K+2M 2M 2M+2M Overheads (virtualized) 4K 4K+4K 4K+2M 2M 2M+2M 4K 4K+4K 4K+2M 2M 2M+2M 1200% 1000% 800% 600% 400% 200% 0% Execution time overhead graph500 memcached NPB:CG gups 4/17/2015 UNIVERSITY OF WISCONSIN 23

24 Cost of virtualization Execution time overhead 100% 80% 60% 40% 20% 0% 202% 126% 4K 4K+4K 4K+2M 2M 2M+2M Overheads (native) 4K 4K+4K 4K+2M 2M 2M+2M Overheads (virtualized) 4K 4K+4K 4K+2M 2M 2M+2M 4K 4K+4K 4K+2M 2M 2M+2M 1200% 1000% 800% 600% 400% 200% 0% Execution time overhead graph500 memcached NPB:CG gups 4/17/2015 UNIVERSITY OF WISCONSIN 24

25 Reducing translation overhead Bhargava et. al: page walk cache Opportunity? PTE reuse (10% of entries cover 90% of accesses) Why? Nested translations are redundant 4/17/2015 UNIVERSITY OF WISCONSIN 25

26 Reducing translation overhead Bhargava et. al: page walk cache Page walk cache Why not cache L1 entries? What is the NTLB? Caches guest physical to system physical Skips the 2 nd dimension walk 4/17/2015 UNIVERSITY OF WISCONSIN 26

27 That was fun, let s make it more complicated DEVICES AND VIRTUAL MEMORY 4/17/2015 UNIVERSITY OF WISCONSIN 27

28 No IOMMU: No Virtualization Proc Kernel Physical Device

29 No IOMMU (virtualized) Proc Kernel Guest Phys VMM Kernel Physical Device

30 Virtualization Devices accessed by physical addresses o Emulation of IO devices is too expensive! Approach 1: VMM driver (paravirtualization) o o Protection domains: IOMMU checks permissions for the memory location; use physical address Need to rewrite drivers! Approach 2: Guest driver (true virtualization) o o Direct Assignment: driver uses guest physical address IOMMU translates to machine physical address

31 IOMMU Overview Memory IOMMU Read memory Device table lookup Address translation service MMU Send interrupts Interrupt remapper VM proc TLB CPU proc VM proc IOTLB GPU I/O Device

32 History Initially combination of o o GART o o DEV o o GART (graphics aperture remapping table) and DEV (device exclusion vector) Physical-to-physical translation so graphics addresses appear contiguous IOMMU is a generalization Devices classified into domains Each domain is allowed to access a set of physical addresses

33 Laundry list of features I/O page tables for I/O devices to access memory o o permission checking virtual address translation Interrupt remapping for I/O interrupts Service page faults from I/O devices Legacy I/O User mode device access VM guest device access Virtualized user mode device access Two-level address translation Interrupt virtualization

34 IOMMU data structures

35 I/O page tables

36 Memory Device memory access PT PT Data Data IOMMU TLB ATS Miss MMU Hit Hit Miss Dev. Table TLB IOTLB CPU GPU I/O Device

37 Page faults (before) Generated if the I/O device accesses unallowed memory Fatal error Written to a log Requires pinned memory

38 Page faults (now) Generated if the I/O device accesses unallowed memory Written to a buffer Interrupt raised on CPU core o (Kernel) driver handles the fault No support to notify the device it should retry o Device keeps on executing/waiting for the TLB miss

Reducing Memory Virtualization Overheads in Virtualized Datacenters. Jayneel Gandhi, Arkaprava Basu, Michael M. Swift, Mark D.

Reducing Memory Virtualization Overheads in Virtualized Datacenters Jayneel Gandhi, Arkaprava Basu, Michael M. Swift, Mark D. Hill Executive Summary Problem: TLB misses in virtual machines Hardware-virtualized