Nested Virtualizationon ARM

Size: px

Start display at page:

Download "Nested Virtualizationon ARM"

Logan Craig
5 years ago
Views:

Nested Virtualizationon ARM NEVE: Nested

Dall Shih-Wei Li Jason Nieh Marc Zyngier LEADING

columbia.edu christoffer.dall@linaro.org shih-wei@cs.

1 Nested Virtualizationon ARM NEVE: Nested Virtualization Extensions Jin Tack Lim Christoffer Dall Shih-Wei Li Jason Nieh Marc Zyngier LEADING COLLABORATION IN THE ARM ECOSYSTEM connect.linaro.org

2 Nested Virtualization Hypervisor Hypervisor Hardware

3 Terminology Nested Nested Guest Hypervisor Host Hypervisor Hardware

4 Use Cases 1. Run guest operating systems with built-in virtualization. 2. IaaS hosting private clouds 3. Test your hypervisor in a 4. Debug your hypervisor in a 5. Develop hypervisors using a cloud

5 ARM Virtualization Extensions EL0 User Space User Space EL1 EL2 Hypervisor

6 ARM Nested Virtualization EL0 User Space User Space EL1 Virtual EL2 Guest Hypervisor Guest Hypervisor EL2 Host Hypervisor

7 ARM Nested Virtualization EL0 User Space User Space EL1 EL?? Guest Hypervisor Guest Hypervisor EL2 Host Hypervisor

8 ARMv8.0 Nested Virtualization EL0 User Space User Space EL1 EL0 Guest Hypervisor Guest Hypervisor Trap-and-emulate EL2 Host Hypervisor

9 ARMv8.0 Nested Virtualization EL0 User Space User Space EL1 EL1 Guest Hypervisor Guest Hypervisor?? -and-emulate EL2 Host Hypervisor

10 ARMv8.3 Nested Virtualization Supports running the guest hypervisor in EL1 EL0 User Space HCR_EL2.NV: EL1 Traps EL2 operations executed in EL1 to EL2 EL1 Guest Hypervisor Traps eret to EL2 Trap -and-emulate CurrentEL reports EL2 even in EL1 EL2 Host Hypervisor

11 K/ARM Nested Virtualization Implementation CPU Virtualization Memory Virtualization Timer Virtualization Interrupt Virtualization

12 Nested CPU Virtualization struct kvm_cpu_context { u64 sys_regs[nr_sys_regs]; + u64 el2_regs[nr_el2_regs]; } struct kvm_vcpu_arch { struct kvm_cpu_context ctxt; }

13 Hypervisor- Switch Host EL0 EL1 EL2 LinuxK Save EL1 sys_regs Restore EL1 sys_regs

14 Hypervisor-Hypervisor Switch Host EL0 EL1 Guest Hypervisor EL2 LinuxK Save/restore EL1 sys_regs Save/restore el2_regs

15 Emulating EL2 in EL1 Define mapping of EL2 registers to EL1 registers Example: TTBR0_EL2 to TTBR0_EL1 Example: SCTLR_EL2 adapted to SCTLR_EL1 Shadow EL1 registers

16 Shadow Registers &sys_regs &shadow_sys_regs PSTATE.mode == EL0/1 PSTATE.mode == EL2 u64 *vcpu->ctxtx.hw_regs

17 Virtual Exceptions Trap to virtual EL2 Forward exceptions EL0 EL1 User Space Emulate virtual exceptions vel2 Guest K EL2 Host K

18 Virtual Exceptions Returning from virtual EL2 Trap eret to EL2 (ARMv8.3) EL0 EL1 User Space Emulate virtual exception return vel2 Guest K EL2 Host K

19 K/ARM Nested Virtualization Implementation CPU Virtualization Memory Virtualization Timer Virtualization Interrupt Virtualization

20 Memory Virtualization EL0 User Space Stage 1: VA -> IPA EL1

21 Memory Virtualization EL0 User Space Stage 1: VA -> IPA EL1 Stage 2: IPA -> PA EL2 Host Hypervisor

22 Memory Virtualization Nested EL0 User Space Stage 1: VA -> IPA EL1???? Guest Hypervisor Stage 2: IPA -> PA EL2 Host Hypervisor

23 Memory Virtualization Nested EL0 User Space Stage 1: VA -> IPA EL1 Virtual stage 2 EL2 Guest Hypervisor Host Hypervisor Shadow Stage 2: IPA -> PA

24 K/ARM Nested Virtualization Implementation CPU Virtualization Memory Virtualization Timer Virtualization Interrupt Virtualization

25 Nested Timer Virtualization ARM provides a virtual and physical timer in EL1 EL2 provides a separate EL2 hyp timer Nested K/ARM supports a virtual CPU with EL2 and the hyp timer

26 K/ARM Nested Virtualization Implementation CPU Virtualization Memory Virtualization Timer Virtualization Interrupt Virtualization

27 ARM Generic Interrupt Controller (GIC) CPU Interface IRQ Device Interrupt Lines Dist. CPU Interface ACK CPU GIC

28 ARM Generic Interrupt Controller (GIC) List Registers (LRs) CPU Interface IRQ Dist. Virtual CPU Interface CPU Interface VIRQ ACK CPU Virtual CPU Interface ACK GIC List Registers (LRs)

29 Nested Interrupt Virtualization Nested User Space Deliver virtual interrupts from the host to the Virtual CPU Interface LRs Guest M Host M

30 Nested Interrupt Virtualization Deliver virtual interrupts from the guest hypervisor to the nested Nested User Space Shadow list registers Virtual CPU Interface The nested can ACK and EOI virtual interrupts without trapping LRs Guest M Host M

31 Performance Evaluation Problem: No ARMv8.3 hardware available. Solution: Use ARMv8.0 hardware with the software modification

32 Emulating v8.3 on v8.0 Nested EL0 EL1 OS Guest Hypervisor SW Modification EL2 Host Hypervisor ARMv8.0 Hardware

33 Hypercall MicroBenchmark Nested EL0 EL0 EL1 OS EL1 EL2 Hypercall OS Hypervisor Return Hypercall Return Guest Hypervisor EL2 Host Hypervisor

34 Hypercall MicroBenchmark ARMv8.3 Nested Cycle counts Ratio to 1 155x

35 Nested Exit/Entry on ARM Nested EL0 EL1 OS Exit Guest Hypervisor Entry EL1 Registers EL2 Host Hypervisor EL2 Registers > 120 traps

36 NEVE: NEsted Virtualization Extensions for ARM Improves performance of nested virtualization Supports unmodified guest hypervisors and OSes Provides two techniques to avoid traps (To appear as an academic paper in SOSP later this month)

37 Register Classification registers: Only affect s execution Hypervisor control registers: Affect hypervisor s execution

38 Registers Nested EL0 EL1 OS Hypercall Guest Hypervisor Return EL2 Host Hypervisor This is when register states are used

39 Registers: Logging to Memory Register Without NEVE Trap! msr x0, TTBR0_EL1 Memory

40 Registers: Logging to Memory Register msr x0, TTBR0_EL1 With NEVE Memory TTBR0_EL1

41 Hypervisor control registers Can t apply the technique for registers They have an immediate impact (EL2 system registers) Key Observation Has guest hypervisor s context EL1 Guest Hypervisor EL1 Registers EL2 Host Hypervisor EL2 Registers

42 Hypervisor control registers Can t apply the technique for registers They have an immediate impact Key Observation Has guest hypervisor s context EL1 Guest Hypervisor EL1 Registers EL2 Host Hypervisor EL2 Registers

43 Hypercall MicroBenchmark ARMv8.3 NEVE Nested Nested Cycle counts Ratio to 155x 34x Trap counts

44 lication Workloads lication Description lication Description Kernbench compile Netperf TCP_RR Network performance Hackbench Scheduler stress Netperf TCP STREAM Network performance SPECjvm2008 Java Runtime Netperf TCP MAERTS Network performance MySQL Database management Apache Web server stress Memcached Key-Value store Nginx Web server stress

45 Experimental Setup Hardware Native//Nested Software APM X-Gene (ARMv8.0) 4-way SMP K on K 8-way SMP 12 GB RAM v GB RAM 10 Gb Ethernet Virt I/O (/nested )

46 Normalized overhead (lower is better) lication Benchmarks ARMv8.3 ARMv8.3 Nested NEVE Nested Kernbench Hackbench SPECjvm2008 TCP RR TCP STREAM TCP MAERTS Apache Nginx Memcached MySQL

47 Normalized overhead (lower is better) lication Benchmarks ARMv8.3 ARMv8.3 Nested NEVE Nested x86 Nested Kernbench Hackbench SPECjvm2008 TCP RR TCP STREAM TCP MAERTS Apache Nginx Memcached MySQL

48 Conclusion We have an implementation of K/ARM for v8.3 Evaluated nested virtualization performance by emulating ARMv8.3 Nested virtualization on ARMv8.3 incurs high overhead Due to the exit multiplication problem NEVE enhances performance significantly by reducing number of traps NEVE is used as basis for extended nested virtualization support in ARMv8.4

49 Code Nested CPU Virtualization patches for ARMv8.3 [RFC v2]: Nested Memory Virtualization patches for ARMv8.3 [RFC]: v8.3 and NEVE Paravirtualization on Linux v4.12-rc1: QEMU Patches:

Nested Virtualization on ARM

Nested Virtualization on ARM NEVE: Nested Virtualization Extensions Jin Tack Lim Christoffer Dall Shih-Wei Li Jason Nieh Marc Zyngier LEADING COLLABORATION IN THE ARM ECOSYSTEM jitack@cs.columbia.edu christoffer.dall@linaro.org