ARM Virtualization: Performance and Architectural Implications. Christoffer Dall, Shih-Wei Li, Jin Tack Lim, Jason Nieh, and Georgios Koloventzos

Transcription

1 ARM Virtualization: Performance and Architectural Implications Christoffer Dall, Shih-Wei Li, Jin Tack Lim, Jason Nieh, and Georgios Koloventzos

2 ARM Servers ARM Network Equipment Virtualization

3 Virtualization Native Virtual Machines Hypervisor Hardware Hardware

4 ARM Hardware Virtualization Support Virtualization Extensions

5 x86 Root (Hypervisor) Non-Root () Exit Save/Restore state to CS

6 ARM Virtualization Extensions EL0 User EL1 Trap EL2 Hypervisor

7 EL2 Controlled by EL2 system registers Limited to support hypervisors, not OS kernels EL0 EL1 EL2 User _EL1 sysregs _EL2 sysregs

8 ARM Virtualization Extensions Design Choices 1. Clear hierarchy from user to kernel to hypervisor 2. Reduced complexity EL0 EL1 EL2 User Hypervisor

9 ARM Virtualization Performance?

10 Measurement Study Micro-benchmarks: low-level hypervisor operations Macro-benchmarks: application workloads

11 Hardware Setup ARM Hardware HP Moonshot m bit ARMv8-A 2.4 GHz APM Atlas CPU 8-way SMP 64 GB RAM (capped at 16 GB) 10 GB Ethernet x86 Hardware Dell PowerEdge r bit x86_x GHz Intel Xeon ES way SMP 16 GB RAM 10 GB Ethernet

12 Software Setup

13 -to-hypervisor Transitions Scheduling Interrupts Hypervisor Memory I/O

14 No-Op Hypercall Hypercall Return Hypervisor

15 Micro Results CPU Clock Cycles ARM x86 K Xen K Xen Hypercall 6, ,300 1,228 1: ARM can be either much faster or slower than x86

16 x86 ARM Non-Root () EL0 User Exit EL1 EL2 Hypervisor Trap Root (Hypervisor)

17 Micro Results CPU Clock Cycles ARM x86 K Xen K Xen Hypercall 6, ,300 1,228 1: ARM can be either much faster or slower than x86 -> x86 Exit more complicated than ARM Trap 2: K is much slower than Xen on ARM

18 Hypervisor Design Type 1 (Bare-Metal) Type 2 (Hosted) Hypervisor OS Hypervisor Hardware Hardware

19 Xen ARM: Bare-Metal EL0 EL1 HVC Ret EL2 Xen

20 K/ARM: Hosted Host EL0 EL1 Linux K EL2 K lowvisor Software World Switch of all CPU state *ASPLOS 2014: K/ARM: The Design and Implementation of the Linux ARM Hypervisor

21 K/ARM: Hosted Host EL0 EL1 Linux K EL2 switch state 3. HVC 2. Ret K lowvisor 1. HVC 4. Ret *ASPLOS 2014: K/ARM: The Design and Implementation of the Linux ARM Hypervisor

22 Micro Results CPU Clock Cycles ARM x86 K Xen K Xen Hypercall 6, ,300 1,228 1: ARM can be either much faster or slower than x86 -> x86 Exit more complicated than ARM Trap 2: K is much slower than Xen on ARM -> ARM architecture not designed for Type 2

23 lication Workloads lication Description Kernbench Hackbench SPECjvm2008 Netperf Apache Memcached MySQL compile Scheduler stress Java workload Network performance Web server stress Key-Value store Database workload

24 lication Performance K ARM Xen ARM K x86 Xen x Kernbench Hackbench SpecJ2008 TCP_RR Normalized overhead (lower is better) TCP_STREAM TCP_MAERTS Apache Memcached MySQL

25 Virtualized I/O I/O Trap Hypervisor Hardware

26 K I/O Model Linux K Hardware

27 Xen Device Drivers : Dom0 Linux Driver Driver Xen Hardware

28 Xen I/O Model Dom0 Switch Trap Xen Copy Data -to- communication Hardware

29 K ARM I/O Xen ARM I/O Trap is slow Trap is fast But all you do is a trap But you do much more

30 Architecture Improvements

31 VHE Virtualization Host Extension

32 K Host EL0 EL1 Linux K EL2 World Switch K lowvisor ~ 6,000 cycles for a hypercall!

33 K + VHE Host EL0 EL1 EL2 Linux K Trap

34 VHE 1. Expand EL2 to support all EL1 features 2. EL1 register accesses to go to EL2

35 VHE Available in ARMv8.1 No (public) hardware yet

36 Conclusions Micro operations: ARM can be faster than x86 Not achievable for Type 2 hypervisors Type 1 is dominated by other I/O costs ARM overhead is comparable to x86 ARMv8.1 adds VHE for hosted hypervisors The software matters!