Performance overhead of KVM on Linux 3.9 on ARM Cortex-A15

Transcription

1 Performance overhead of KVM on Linux 3.9 on ARM Cortex-A15 Lars Rasmusson 1,2, Diarmuid Corcoran 1 1 Ericsson AB, 2 SICS

2 Context : Where

3 Context : Where Embedded Domain Devices Access Core

4 Context : Why The nature of a typical embedded system is changing. Instead of a single highly coherent function we foresee several orthogonal application sharing the same HW resources. It seem likely that ARM architecture will be a dominant player in the embedded space. The virtualization extension to ARM architecture are new and we could find no existing existing measurements or experiments. It seems like linux will dominate the embedded space. We looked at several options in terms of linux and hypervisors. After a survey it seem clear that ARM/KVM will dominate. That led us to these experiments to understand the characteristics of ARM/KVM. It was not our intention to dig deep into the reasons, this is an ongoing topic.

5 KVM KVM exposes /dev/kvm create VM alloc memory read/write virt. CPU regs inject interrupt run a virt. CPU Kivity, et al. (2007)

6 ARM Cortex A15 Multicore and Platform

7 KVM/ARM system Dall & Nieh, (2013)

8 KVM/ARM system Traps and IRQs go either to the guest kernel, or to the host via Hyp mode syscall trap virt IRQ No traps to Hyp when read/write: Stage-1 page tables virtual GIC virtual timers/counters HW IRQ trap IP IRQ/ distributor Dall & Nieh, (2013)

9 KVM/ARM system = VM HOST VM = Just switching mode: VM Hyp = MMIO load op in guest = as above, but QEmu emulated = from signalling until response = ackn. and complete interrupt Dall & Nieh, (2013)

10

11 Starring: /dev/zero dd kvmbox qemu virtio network driver virtio Gbit Ethernet dd stbox /dev/null vebox port forw. smsc911x network driver kvmbox - KVM ARM virtual machine with 2 virtual Cortex A15 SMP cores vebox - real VersatileExpress with 2 Cortex A15 (and 3 disabled A7) cores stbox - x86_64 machine with 24 cores and Gb Ethernet

12 Send 100 MB via network Listener: nc -lk <port> >/dev/null Sender: dd if=/dev/zero bs=1m count=100 nc <dest> <port> Sender Listener kvmbox user kvmbox virtio kvmbox user kvmbox virtio vebox stbox vebox Unit: MB/s as reported by dd kvm's user mode networking uses virtio and port forwarding to kvmbox host and guest have 2 cores and smp=2 stbox between stbox and stbox2 on LAN: 11.9 MB/s

13 Observations Virtio looks like it produces very little overhead: 82 MB/s compared to the host s 85 MB/s. User networking less impressive with 50.8 MB/s. (but...) sending from vebox to kvmbox is about twice as fast as sending from kvmbox to vebox. When sending across the network, virtio is slightly faster than user networking.

14 Sending over the network Unit: MB/s (Example of how to read the table: when host (vebox) is pinned to core 1 and kvm (kvmbox) is pinned to core 0, and both kvm and host are sending concurrently, then kvm sends to localhost at 46.1 MB/s.) { (over the network) see second next slide

15 Observations KVM to localhost is always around 51 MB/s. The good result from the first experiment appears to be a fluke. When sending concurrently to localhost, kvm has no overhead compared to when pinned to different cores, but 20 and 35 percent overhead when pinned to same core.... and the host throughput goes down from 77 to 46, even if host and guest are pinned to different cores Some bottleneck (cpu, shared data structure,?) seems to exist.

16 Observations Column Concurrent to vebox shows the overhead of using virtio to send data from kvm to vebox. Kvmbox performance drops 70 percent, (46 13 MB/s)

17 Sending over the network... when both host and guest are not and are active at the same time

18 Observations When only one is sending, KVM througput is about 85% of host. When both host and guest are sending, throughput varies wildly. Performance depends on which cores host and guest are pinned to (!) Sometimes kvm outperforms the host. kvm s result is worst when both host and kvm are sending and both are pinned to both cores

19 Disk read speed hdparm reads buffered and unbuffered from the SD disk. kvm guest unbuffered becomes buffered because the host file system buffers pages from the disk image. hdparm -Tt <dev> vebox Unit: MB/s as reported by hdparm losetup to a file with 10M zeros backed on sda, vda, or nfs. buffered reads get faster on consecutive runs

20 Observations We tested reading from the physical SDRAM card /dev/ sda1, and from a loopback-mounted file on /dev/sda1, and a loopback-mounted file on an external nfs file system. Reading from the linux file buffer cache has about 7 percent overhead for KVM. Even though buffered reads should write through to the device, there seems to be caching effects when reading from /dev/vda, and from /dev/loop1. hdparam does not write through completely, and thus reports incorrect values.

21 Geekbench2 Geekbench2 is a collections of benchmarks for many platforms and CPUs that tries to compute an average score for a platform. VEBOX KVMBOX vebox kvmbox factor Overall Score Integer blowfish, compress Floating Point LU-decomp, sharpen image Memory stdlib allocate, stdlib write Stream stream copy, stream scale

22 Observations The integer and floating point exercises have a large unexplained overhead: 32 percent. They ought not be affected by the virtualization. (ARM supports optional extensions like NEON (mandatory for A15), which can be software emulated. Perhaps that is what we are seeing here.)

23 Next Repeat experiments on alternative ARM hardware. Profile ARM/KVM under load on a many core system understand the overhead of the hierarchical scheduling real-time behaviour, RT-PREEMPT Locate bottlenecks Understand how to achieve and improve performance

24 Summary Hyp-mode KVM on ARM in Linux since 3.9 Virtio supported, and is faster than simple emulated networking. Quite unpredictable network performance while running multiple concurrent processes and VMs, otherwise about 15 percent slowdown compared to native. Unexplained slowdowns were noticed with geekbench2 on some compute heavy workloads. On VersatileExpress, performance varies across cores.