Networks and Opera/ng Systems Chapter 21: Virtual Machine Monitors ( )

Size: px

Start display at page:

Download "Networks and Opera/ng Systems Chapter 21: Virtual Machine Monitors ( )"

Chastity Garrison
6 years ago
Views:

1 Networks and Opera/ng Systems Chapter 21: Virtual Machine Monitors ( ) Donald Kossmann & Torsten Hoefler Frühjahrssemester 2013 Systems Group Department of Computer Science ETH Zürich

2 Last /me: I/O Network stack implementa/on Network devices and network I/O Memory management in the I/O subsystem Performance issues Buffering Mul/ple queues and receive side scaling

3 This /me: Virtual Machine Monitors Basic defini/ons Why would you want one? Structure How does it work? CPU MMU Memory Devices Network Acknowledgement: Thanks to Steve Hand for some of the slides!

4 What is a Virtual Machine Monitor? Virtualizes an en/re (hardware) machine Contrast with OS processes Interface provided is illusion of real hardware Applica/ons are therefore complete Opera/ng Systems themselves Terminology: Guest Opera+ng Systems Old idea: IBM VM/CMS (1960s) Recently revived: VMware, Xen, Hyper V, kvm, etc.

system VMM Hypervisor Some folks dis/nguish the Virtual

5 VMMs and Hypervisors Creates illusion of hardware App App Guest opera/ng system VMM App App Guest opera/ng system VMM Hypervisor Some folks dis/nguish the Virtual Machine Monitor from the Hypervisor (we won t) Real hardware

6 Why would you want one? Diagrams: Server consolida/on (program assumes own machine) Performance isola/on Backward compa/bility Cloud compu/ng (unit of selling cycles) Something under the OS: replay, audi/ng, trusted compu/ng, rootkits

Running mul/ple OSes on one machine App App App App App App Applica/on compa/bility I use Ubuntu for almost everything, but I edit slides in PowerPoint Some people compile

7 Running mul/ple OSes on one machine App App App App App App Applica/on compa/bility I use Ubuntu for almost everything, but I edit slides in PowerPoint Some people compile Barrelfish in a Debian VM over Windows 7 with Hyper V Hypervisor Real hardware Backward compa/bility Nothing beats a Windows 98 virtual machine for playing old computer games

8 Server consolida/on Applica/on Applica/on Hypervisor Real hardware Applica/on Many applica/ons assume they have the machine to themselves Each machine is mostly idle Consolidate servers onto a single physical machine

9 Resource isola/on Applica/on Applica/on Hypervisor Real hardware Applica/on Surprisingly, modern OSes do not have an abstrac/on for a single applica/on Performance isola/on can be cri/cal in some enterprises Use virtual machines as resource containers

10 Cloud compu/ng Applica/on Applica/on Applica/on Hypervisor Real hardware Applica/on Applica/on Hypervisor Real hardware Applica/on Applica/on Hypervisor Real hardware Applica/on Applica/on Hypervisor Real hardware Applica/on Applica/on Hypervisor Real hardware Applica/on Hypervisor Real hardware Selling compu/ng capacity on demand E.g. Amazon EC2, GoGrid, etc. Hypervisors decouple alloca+on of resources (VMs) from provisioning of infrastructure (physical machines)

11 Opera/ng System development Compiler Editor Visual Studio Hypervisor Real hardware Building and tes/ng a new OS without needing to reboot real hardware VMM omen gives you more informa/on about faults than real hardware anyway

12 Other cool applica/ons Tracer Applica/on Applica/on Hypervisor Real hardware Tracing Debugging Execu/on replay Lock step execu/on Live migra/on Rollback Specula/on Etc.

13 How does it all work? Note: a hypervisor is basically an OS With an unusual API Many func/ons quite similar: Mul/plexing resources Scheduling, virtual memory, device drivers Different: Crea/ng the illusion of hardware to applica/ons Guest OSes are less flexible in resource requirements

14 Hosted VMMs Applica/on Applica/on App App Guest opera/ng system Examples: VMware worksta/on Linux KVM Microsom Hyper V VMM Host opera/ng system Real hardware

15 Hypervisor based VMMs Console Mgmt. Console (Mgmt) opera/ng system App App Guest opera/ng system App App Guest opera/ng system Examples: VMware ESX IBM VM/CMS Xen VMM VMM VMM Hypervisor Real hardware

16 How to virtualize The CPU (s)? The MMU? Physical memory? Devices (disks, etc.)? The Network and?

17 Virtualizing the CPU A CPU architecture is strictly virtualizable if it can be perfectly emulated over itself, with all nonprivileged instruc/ons executed na/vely Privileged instruc/ons trap Kernel mode (i.e. the VMM) emulates instruc/on Guest s kernel mode is actually user mode Or another, extra privilege level (such as ring 1) Examples: IBM S/390, Alpha, PowerPC

18 Virtualizing the CPU A strictly virtualizable processor can execute a complete na/ve Guest OS Guest applica/ons run in user mode as before Guest kernel works exactly as before Problem: x86 architecture is not virtualizable About 20 instruc/ons are sensi/ve but not privileged Mostly segment loads and processor flag manipula/on

19 Non virtualizable x86: example PUSHF/POPF instruc/ons Push/pop condi/on code register Includes interrupt enable flag (IF) Unprivileged instruc/ons: fine in user space! IF is ignored by POPF in user mode, not in kernel mode VMM can t determine if Guest OS wants interrrupts disabled! Can t cause a trap on a (privileged) POPF Prevents correct func/oning of the Guest OS

20 Solu/ons 1. Emula/on: emulate all kernel mode code in somware Very slow par/cularly for I/O intensive workloads Used by, e.g., SomPC 2. Paravirtualiza8on: modify Guest OS kernel Replace with explicit trap instruc/on to VMM Also called a HyperCall (used for all kinds of things) Used by, e.g., Xen 3. Binary rewri/ng: Protect kernel instruc/on pages, trap to VMM on first IFetch Scan page for POPF instruc/ons and replace Restart instruc/on in Guest OS and con/nue Used by, e.g. VMware 4. Hardware support: Intel VT x, AMD V Extra processor mode causes POPF to trap

21 Virtualizing the MMU Hypervisor allocates memory to VMs Guest assumes control over all physical memory VMM can t let Guest OS to install mappings Defini/ons needed: Virtual address: a virtual address in the guest Physical address: as seen by the guest Machine address: real physical address As seen by the Hypervisor

22 Virtual/Physical/Machine Guest Virtual AS Guest Physical AS Machine Memory Guest 1: Guest 2: 5

23 MMU Virtualiza/on Cri/cal for performance, challenging to make fast, especially SMP Hot unplug unnecessary virtual CPUs Use mul/cast TLB flush paravirtualiza/ons etc Xen supports 3 MMU virtualiza/on modes 1. Direct ( Writable ) pagetables 2. Shadow pagetables 3. Hardware Assisted Paging OS Paravirtualiza/on compulsory for #1, op/onal (and very beneficial) for #2&3

24 Paravirtualiza/on approach Guest OS creates page tables the hardware uses VMM must validate all updates to page tables Requires modifica/ons to Guest OS Not quite enough VMM must check all writes to PTEs Write protect all PTEs to the Guest kernel Add a HyperCall to update PTEs Batch updates to avoid trap overhead OS is now aware of machine addresses Significant overhead!

25 Para Virtualizing the MMU Guest OSes allocate and manage own PTs Hypercall to change PT base VMM must validate PT updates before use Allows incremental updates, avoids revalida/on Valida/on rules applied to each PTE: 1. Guest may only map pages it owns* 2. Pagetable pages may only be mapped RO VMM traps PTE updates and emulates, or unhooks PTE page for bulk updates

26 Writeable Page Tables : 1 Write fault guest reads first guest write Virtual Machine Guest OS page fault VMM MMU Hardware

27 Writeable Page Tables : 2 Emulate? guest reads first guest write Virtual Machine Guest OS yes emulate? VMM MMU Hardware

28 Writeable Page Tables : 3 Unhook guest reads guest writes X Virtual Machine Guest OS VMM MMU Hardware

29 Writeable Page Tables : 4 First Use guest reads guest writes X Virtual Machine Guest OS page fault VMM MMU Hardware

30 Writeable Page Tables : 5 Re hook guest reads guest writes Virtual Machine Guest OS validate VMM MMU Hardware

31 Writeable page tables require paravirtualiza/on Guest Virtual AS Machine Memory Guest 1: Guests directly share Machine Memory 9 6 Guest 2: 5

32 Shadow Page Tables Guest OS sets up its own page tables Not used by the hardware! VMM maintains shadow page tables Map directly from Guest VAs to Machine Addresses Hardware switched whenever Guest reloads PTPR VMM must keep V M table consistent with Guest V P table and it s own P M table VMM write protects all guest page tables Write trap: apply write to shadow table as well Significant overhead!

33 Shadow Page Tables Guest Virtual AS Guest Physical AS Machine Memory Guest 1: Shadow page table mappings 9 6 Guest 2: 5

34 Shadow page tables guest reads Virtual Guest Physical guest writes Guest changes op/onal, but help with batching, knowing when to unshadow accessed and dirty bits Latest algorithms work remarkably well updates Virtual Machine Guest OS VMM Hardware MMU

35 Hardware support Nested page tables Rela/vely new in AMD (NPT) and Intel (EPT) hardware Two level transla/on of addresses in the MMU Hardware knows about: V P tables (in the Guest) P M tables (in the Hypervisor) Tagged TLBs to avoid expensive flush on a VM entry/ exit Very nice and easy to code to One reason kvm is so small Significant performance overhead

36 Memory alloca/on Guest OS is not expec/ng physical memory to change in size! Two problems: Hypervisor wants to overcommit RAM How to reallocate (machine) memory between VMs Phenomenon: Double Paging Hypervisor pages out memory GuestOS decides to page out physical frame (Unwivngly) faults it in via the Hypervisor, only to write it out again

37 Ballooning Technique to reclaim memory from a Guest Install a balloon driver in Guest kernel Can allocate and free kernel physical memory Just like any other part of the kernel Uses HyperCalls to return frames to the Hypervisor, and have them returned Guest OS is unware, simply allocates physical memory

38 Ballooning: taking RAM away from a VM Guest physical address space Balloon Balloon driver 1. VMM asks balloon driver for memory 2. Balloon driver asks Guest OS kernel for more frames inflates the balloon 3. Balloon driver sends physical frame numbers to VMM 4. VMM translates into machine address and claims the frames

39 Ballooning: taking RAM away from a VM Guest physical address space Physical memory claimed by balloon driver Balloon Balloon driver 1. VMM asks balloon driver for memory 2. Balloon driver asks Guest OS kernel for more frames inflates the balloon 3. Balloon driver sends physical frame numbers to VMM 4. VMM translates into machine addresses and claims the frames

40 Returning RAM to a VM Guest physical address space Balloon Balloon driver 1. VMM converts machine address into a physical address previously allocated by the balloon driver 2. VMM hands PFN to balloon driver 3. Balloon driver frees physical frame back to Guest OS kernel deflates the balloon

41 Virtualizing Devices Familiar by now: trap and emulate I/O space traps Protect memory and trap Device model : somware model of device in VMM Interrupts upcalls to Guest OS Emulate interrupt controller (APIC) in Guest Emulate DMA with copy into Guest PAS Significant performance overhead!

42 Paravirtualized devices Fake device drivers which communicate efficiently with VMM via hypercalls Used for block devices like disk controllers Network interfaces VMware tools is mostly about these Drama/cally beyer performance!

43 Networking Virtual network device in the Guest VM Hypervisor implements a som switch En/re virtual IP/Ethernet network on a machine Many different addressing op/ons Separate IP addresses Separate MAC addresses NAT Etc.

44 Where are the real drivers? 1. In the Hypervisor E.g. VMware ESX Problem: need to rewrite device drivers (new OS) 2. In the console OS Export virtual devices to other VMs 3. In driver domains Map hardware directly into a trusted VM Device Passthrough Run your favorite OS just for the device driver Use IOMMU hardware to protect other memory from driver VM 4. Use self virtualizing devices

45 Xen 3.x Architecture VM0 Device Manager & Control s/w VM1 Unmodified User Somware VM2 Unmodified User Somware VM3 Unmodified User Somware GuestOS (XenLinux) Virtual switch GuestOS (XenLinux) SMP GuestOS (XenLinux) Unmodified GuestOS (WinXP) Na/ve Device Drivers Front End Device Drivers Front End Device Drivers Front End Device Drivers Control IF Safe HW IF Event Channel Virtual CPU Xen Virtual Machine Monitor Virtual MMU Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE) Thanks to Steve Hand for some of these diagrams

46 Remember this card?

47 SR IOV Single Root I/O Virtualiza/on Key idea: dynamically create new PCIe devices Physical Func/on (PF): original device, full func/onality Virtual Func/on (VF): extra device, limited fun/onality VFs created/destroyed via PF registers For networking: Par//ons a network card s resources With direct assignment can implement passthrough

func/on Virtual func/on Virtual func/on Physical func/on

48 SR IOV in ac/on VM VM VM VM VM VSwitch VF driver VF driver VF driver VNIC drvr PF driver VMM IOMMU PCIe Virtual func/on Virtual func/on Virtual func/on Physical func/on Virtual ethernet bridge/switch, packet classifier SR IOV NIC LAN

49 Self virtualizing devices Can dynamically create up to 2048 dis/nct PCI devices on demand! Hypervisor can create a virtual NIC for each VM Somswitch driver programs master NIC to demux packets to each virtual NIC PCI bus is virtualized in each VM Each Guest OS appears to have real NIC, talks direct to the real hardware

50 Next Week Reliable storage OS Research/Future

Background. IBM sold expensive mainframes to large organiza<ons. Monitor sits between one or more OSes and HW

Background. IBM sold expensive mainframes to large organiza<ons. Monitor sits between one or more OSes and HW Virtual Machines Background IBM sold expensive mainframes to large organiza