Lecture 5. KVM for ARM. Christoffer Dall and Jason Nieh. 5 November, Operating Systems Practical. OSP Lecture 5, KVM for ARM 1/42

Size: px

Start display at page:

Download "Lecture 5. KVM for ARM. Christoffer Dall and Jason Nieh. 5 November, Operating Systems Practical. OSP Lecture 5, KVM for ARM 1/42"

Victor Greene
6 years ago
Views:

1 Lecture 5 KVM for ARM Christoffer Dall and Jason Nieh Operating Systems Practical 5 November, 2014 OSP Lecture 5, KVM for ARM 1/42

2 Contents Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 2/42

3 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 3/42

4 Virtualization: reminder Virtual Machine (VM) Popek and Goldberg (1974) A virtual computer system is a hardware-software duplicate of a real existing computer system in which a statistically dominant subset of the virtual processor s instructions execute on the host processor in native mode. OSP Lecture 5, KVM for ARM 4/42

5 Virtualization: reminder (2) Duplicate underlying hardware Virtualization emulation Run (almost) as fast as the physical machine Abstract hardware resources CPU Memory I/O devices OSP Lecture 5, KVM for ARM 5/42

6 Virtualization: reminder (3) Hypervisor, Virtual Machine Monitor (VMM) The two often get confused in literature One hypervisor per physical machine One VMM per VM OSP Lecture 5, KVM for ARM 6/42

7 Virtualization: reminder (4) Virtualization architectures Type I: Baremetal or native Type II: Hosted Source: OSP Lecture 5, KVM for ARM 7/42

8 Approaches to virtualization Binary translation Translate sensitive instructions to non-sensitive ones Can run unmodified guest OSes Paravirtualization gr. para (alongside) + virtualization More efficient than binary translation Also more intrusive: requires modification of guest OS Hardware-assisted virtualization Extend CPU with special hypervisor state Trap-and-emulate Similar for memory, I/O OSP Lecture 5, KVM for ARM 8/42

9 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 9/42

10 KVM Kernel-based Virtual Machine Module for the Linux kernel that turns it into a hypervisor Part of the Linux mainline kernel (since ) Exposes /dev/kvm to user space Requires a user space host to run e.g. QEMU OSP Lecture 5, KVM for ARM 10/42

11 KVM/QEMU on x86 kvm.ko module provides virtualization interface via /dev/kvm QEMU sets up the guest VM using /dev/kvm This requires hardware assisted virtualization (VT-x or AMD-V) Otherwise QEMU will fall back to software emulation KVM architecture is strongly dependent on Linux/x86 OSP Lecture 5, KVM for ARM 11/42

12 Virtualization on x86 (Intel VT-x) AMD-V Secure Virtual Machine looks similarly OSP Lecture 5, KVM for ARM 12/42

13 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 13/42

14 ARMv7-A Instruction Set Architecture Base ISA is not virtualizable CPU modes/privilege Levels: User mode (PL0) Supervisor mode (PL1) Monitor mode (Secure PL1)... OSP Lecture 5, KVM for ARM 14/42

15 ARM TrustZone TrustZone/Secure Monitor does not provide virtualization No support for trap-and-emulate logic Secure boot, chain of trust Memory isolation from a rich OS Running trusted/proprietary software (e.g. Digital Rights Management) OSP Lecture 5, KVM for ARM 15/42

16 ARM Virtualization Extensions Extensions for CPU and MMU virtualization Implemented in Cortex-A15, Cortex-A7 New processor mode: Hyp mode (PL3) OSP Lecture 5, KVM for ARM 16/42

17 ARM VE: CPU virtualization OSP Lecture 5, KVM for ARM 17/42

18 ARM VE: CPU virtualization (2) Traps are configurable Sensitive instructions may trap to Supervisor mode Hyp mode Hyp mode has a reduced number of registers compared to Supervisor mode OSP Lecture 5, KVM for ARM 18/42

19 ARM VE: Memory virtualization Guest physical Hypervisor Virtual Memory Hypervisor Machine memory OSP Lecture 5, KVM for ARM 19/42

20 ARM VE: Memory virtualization (2) MMU supports two stages of translation: Stage 1: Virtual (VA) to Intermediate Physical (IPA) Managed by guest OS Stage 2: Intermediate Physical (IPA) to Host Physical (PA) Managed by hypervisor OSP Lecture 5, KVM for ARM 20/42

21 ARM VE: I/O virtualization Interrupts Generic Interrupt Controller (GIC) VGIC Memory-mapped interface (MMIO) Inter-Processor Interrupts (IPI) Timers Generic Timer Architecture Virtual counter, virtual timer OSP Lecture 5, KVM for ARM 21/42

22 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 22/42

23 Intel VT-x: CPU virtualization OSP Lecture 5, KVM for ARM 23/42

24 KVM on ARM: Challenges KVM architecture is strongly dependent on Linux/x86 Difficult to migrate Linux to Hyp mode Modify a lot of code Maintain separate branch Don t get included in upstream Hyp mode lacks the features of VMX root mode Difficult run an entire OS there OSP Lecture 5, KVM for ARM 24/42

25 Split-mode virtualization Factor KVM into Highvisor and Lowvisor OSP Lecture 5, KVM for ARM 25/42

26 Split-mode virtualization Lowvisor Switches between Host (Highvisor) and VMs Highvisor Leverages KVM/Linux s mechanisms Manages both its page tables and the Lowvisor s Note: trapping to the Highvisor requires double-switching OSP Lecture 5, KVM for ARM 26/42

27 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 27/42

28 World switches VMs and the Host/Highvisor run at the same level of privilege All world switches are intermediated by the Lowvisor The Host controls switching policy OSP Lecture 5, KVM for ARM 28/42

29 VM and Host state Action Nr. State 38 General Purpose (GP) Registers 26 Control Registers 16 VGIC Control Registers Context Switch 4 VGIC List Registers 2 Arch. Timer Control Registers bit VFP registers 4 32-bit VFP Control Registers - CP14 Trace Registers - WFI Instructions - SMC Instructions Trap-and-Emulate - ACTLR Access - Cache ops. by Set/Way - L2CTLR / L2ECTLR Registers OSP Lecture 5, KVM for ARM 29/42

30 Switching from Host to VM 1. Store Host state 2. Configure and load VM state 3. Enable traps from Supervisor to Hyp 4. Enable Stage 2 translation 5. Trap to Supervisor or User OSP Lecture 5, KVM for ARM 30/42

31 Switching from VM to Host 1. Store VM state 2. Configure and load VM state 3. Disable (most) traps from Supervisor to Hyp 4. Disable Stage 2 translation 5. Trap to Host OSP Lecture 5, KVM for ARM 31/42

32 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 32/42

33 Memory virtualization Provided by enabling Stage 2 translation Stage 2 page tables are managed by the Highvisor Stage 2 translation is enabled/disabled by the Lowvisor OSP Lecture 5, KVM for ARM 33/42

34 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 34/42

35 I/O virtualization Direct access to I/O memory is disabled Accesses by the VM will result in Stage 2 pagefaults... and trap to the hypervisor Device access policy is controlled by QEMU Pass-through devices are an exception OSP Lecture 5, KVM for ARM 35/42

36 Interrupt virtualization Interrupts are emulated through VGIC VGIC state is saved/restore on context switches GIC distributor accesses will trap to the hypervisor Virtual distributor routes IPIs OSP Lecture 5, KVM for ARM 36/42

37 Timer virtualization Uses virtual timers Only Hyp mode can access physical timers Virtual timer interrupts trap to hypervisor Hypervisor forwards interrupts to VMs Hypervisor performs ACK and EOI operations Per-CPU timers are multiplexed using the Host s software timers OSP Lecture 5, KVM for ARM 37/42

38 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 38/42

39 Keywords virtualization hardware-assisted virtualization hypervisor virtual machine monitor split-mode virtualization highvisor lowvisor world switch OSP Lecture 5, KVM for ARM 39/42

40 Resources ols2010-pages pdf wpid-asplos2014-kvm.pdf OSP Lecture 5, KVM for ARM 40/42

41 Outline Virtualization KVM Virtualization on ARM KVM/ARM: System architecture KVM/ARM: CPU virtualization KVM/ARM: Memory virtualization KVM/ARM: I/O virtualization Keywords Questions OSP Lecture 5, KVM for ARM 41/42

42 Questions? OSP Lecture 5, KVM for ARM 42/42

KVM/ARM: The Design and Implementation of the Linux ARM Hypervisor

KVM/ARM: The Design and Implementation of the Linux ARM Hypervisor Christoffer Dall Department of Computer Science Columbia University cdall@cs.columbia.edu Jason Nieh Department of Compouter Science Columbia