KVM PERFORMANCE OPTIMIZATIONS INTERNALS. Rik van Riel Sr Software Engineer, Red Hat Inc. Thu May

Transcription

1

2 KVM PERFORMANCE OPTIMIZATIONS INTERNALS Rik van Riel Sr Software Engineer, Red Hat Inc. Thu May

3 KVM performance optimizations What is virtualization performance? Optimizations in RHEL 6.0 Selected RHEL 6.1 features Vhost-net KSM & Transparent Huge Pages Future KVM optimizations Pause Loop Exiting Free Page Hinting & Dynamic Memory Resizing Conclusions

4 What is virtualization performance? Different users have different goals: Throughput or latency on one virtual machine Aggregate throughput of a set of virtual machines Largest number of virtual machines on a system, with acceptable performance (density). KVM in RHEL and RHEV has performance optimizations for all of these goals.

5 RHEL 6.0 KVM optimizations Large hardware scalability 64 VCPU / 1TB memory guests X2apic Lockless RCU synchronization Transparent Huge Pages (THP) Kernel Samepage Merging (KSM) Do not use ticket spinlock inside guest

6 RHEL 6.0 networking (before) KVM started as QEMU acceleration Move perf critical bits to kernel Guest VM QEMU network emulation Network packets copied HW -> host kernel Host kernel -> qemu Qemu -> guest Copy to/from user overhead QEMU kernel/hv tap bridge t x vnic r x NIC

7 RHEL 6.1 vhost-net Virtio network emulation in kernel Eliminates userspace copies Improves latency & throughput Reduces overhead In the future Macvtap Bridge & tap in one Has some limitations Great for VEPA or VN-Link QEMU kernel/hv tap bridge NIC Guest VM t x vhost r x

8 RHEL 6.1 KVM network performance Guest Scale Out RX Vhost vs Virtio - % Host CPU Mbit per % CPU netperf T CP_ST REAM Mbit / % CPU (bigger is bet t er) Message Size (Byt es) Vho st Virt io

9 Kernel Samepage Merging (KSM) Share identical pages Libraries Executables Images Free pages (Windows) 4kB pages only Ksmd scans memory G1 G2 CPU / space tradeoff Kernel

10 KSM & Transparent Huge Pages KSM in 6.0 only scans 4kB pages Transparent Huge Pages (THP) memory excluded THP memory broken into small pages at swap time Too late KSM in 6.1 looks inside huge pages If shareable 4kB page found... break up huge page into small pages... and share those Only done when free memory gets low

11 KSM tuning Automatically done by ksmtuned Ksmd started when free memory low <10% or 2GB free memory Configurable in /etc/ksmtuned.conf KSM uses CPU time, to save memory Good for packing guests onto systems (density) Can help avoid or reduce swap IO If CPU is bottleneck and memory is plentyful... disable KSM

12 Pause Loop Exiting guest spinlocks VCPU 2 holds a spinlock VCPU 1 wants the spinlock, spins for it After VCPU 2 releases the lock, VCPU 1 grabs it VCPU 1 VCPU 2

13 Pause Loop Exiting the problem But what if VCPU 2 is not running? Page fault, host scheduler, KVM/qemu lock,... VCPU 1 could spin for a long time Waste lots of CPU time Could keep VCPU 2 from running! VCPU 1 VCPU 2

14 Pause Loop Exiting a hardware solution Hardware detection of a spinning VCPU Intel Pause Loop Exiting (PLE) AMD Pause Filtering (PF) When a VCPU spins for too long Detect it in hardware and trap to the host Host does something... Problem solved!

15 Pause Loop Exiting directed yield Host side solution is two-fold Waste little time spinning Get lock holder to run ASAP, it may release the lock Spinner and lock holder could run on different CPUs Implement CFS yield_to() to temporarily boost lock holder's priority Guest with lock contention should not lose CPU time Reimplement CFS yield() to temporarily give up CPU time, without losing priority

16 PLE benchmark results AMQP benchmark with 8 and 16 byte messages 4 VCPU guest on 4 physical CPUs No, 2 or 4 VCPU load for no, 1.5x and 2x overcommit 10-25% speedup at high CPU overcommit ratios AMQP 8 byte AMQP 16 byte no PLE PLE no PLE PLE no 1.5x 2.0x 0 no 1.5x 2.0x

17 Free Page Hinting Nested LRU problem Sum of guest memory exceeds host memory Host swaps out least used pages Guest frees least used pages Guest re-uses least used pages first Combined guest & host swapping can lead to terrible performance

18 Nested LRU problem illustrated Three guests HOST HOST MEMORY MEMORY Host is swapping Swap IO due to free memory Guests re-use free pages for new content Free memory contains no useful data Swap IO unnecessary GUEST MEMORY USED FREE SWAP SWAP

19 Free page hinting Free pages contain no useful information The host can throw away free guest pages! Swapout, KSM, etc Give the guest a fresh page on use Skip disk IO on swapin Guest needs to inform host what memory is free Can use a big bitmap Be careful at state transitions (free->used)

20 Nested LRU with page hinting Most free pages eliminated HOST HOST MEMORY MEMORY Guests now fit in RAM The used bits... Swap IO greatly reduced But what if sum of used memory exceeds RAM size? SWAP SWAP GUEST MEMORY USED FREE

21 Dynamic memory resizing We can do more than just eliminate free memory Guest has working set, page cache and free When the host is near swapout It can ask each guest to free some memory Do not shrink guest below minimum size When a guest is near swapout It can ask the host for some memory back Do not grow guest above maximum size

22 Dynamic memory resizing Ask guests to free extra HOST HOST MEMORY MEMORY memory when host needs Obey min/max guest size Fit more guests per host Combine with cgroups for guest prioritization Avoid most swap IO SWAP SWAP GUEST MEMORY USED CACHE FREE

23 Conclusions KVM on RHEL/RHEV the top virt performer 3 of the top 5 SPECvirt scores, including #1 (RHEL 6.0) RHEL 6.1 even faster, more optimizations ahead Keep KVM the best performing virt platform For single guests, aggregate throughput and density Use new hardware features like SR-IOV and PLE Smart resource management More predictable performance, with more guests Cheaper virtualization hosting

24 More information KVM project Red Hat kbase, tech briefs, etc Linux Weekly News

25