KVM Weather Report. Red Hat Author Gleb Natapov May 29, 2013

Size: px

Start display at page:

Download "KVM Weather Report. Red Hat Author Gleb Natapov May 29, 2013"

Veronica Drusilla Gallagher
6 years ago
Views:

1 KVM Weather Report Red Hat Author Gleb Natapov May 29, 2013

2 Part I What is KVM

3 Section 1 KVM Features

4 KVM Features 4 KVM features VT-x/AMD-V (hardware virtualization) EPT/NPT (two dimensional paging) CPU/memory overcommit Scalability (160 vcpus and 2TB RAM are tested) Live Migration KSM Transparent huge pages Security and isolation with svirt and seccomp Secure PCI device assignment using IOMMU SVVP (Server Virtualization Validation Program) PV IO using VirtIO WHQL VirtIO drivers

5 Section 2 KVM Architecture

6 KVM Architecture 6 KVM is a Linux Subsystem

7 KVM VCPU Loop KVM Architecture 7

8 KVM Architecture 8 Network Architecture Emulated Devices E1000 RTL8139 Native drivers Compatibility over performance VirtIO Devices Paravirtualized virtio-net Performace over compatibility Device Assignment Native drivers Compatibility and Performance No migration

9 Block Architecture KVM Architecture 9

10 Section 3 Linux as a Hypervisor

11 Linux as a Hypervisor 11 Why Linux? Scalable scheduler Sophisticated memory management (NUMA/huge pages) Hardware enablement for free Scalable I/O stack (AIO capabilities are lacking) Isolation (cgroups) Security (seccomp/selinux) Tracing and performance monitoring And more...

12 Part II What s new

13 Section 4 APIC Virtualization and Virtual Interrupts

14 APIC Virtualization and Virtual Interrupts 14 Problem: APIC Emulation is Inefficient On each APIC access: #VMEXIT Instruction emulation (x2apic mitigates this) On each interrupt injection: #VMEXIT APIC state evaluation

15 APIC Virtualization and Virtual Interrupts 15 Solution: Move APIC virtualization into CPU! In three easy steps: APIC register virtualization Virtual interrupt delivery Posted interrupt processing

16 APIC Virtualization and Virtual Interrupts 16 APIC Register Virtualization APIC-read reads from vapic page w/o causing #VMEXIT APIC-write writes into vapic page w/o causing #VMEXIT New APIC-write #VMEXIT - Trap like #VMEXIT for APIC register writes that should be handled by VMM (no emulation needed)

17 APIC Virtualization and Virtual Interrupts 17 Virtual Interrupt Delivery New VMCS field Guest interrupt status with two subfields: - Requesting virtual interrupt (RVI) - Servicing virtual interrupt (SVI) Pending interrupt are evaluated on: - VM Entry - TPR access - EOI access - self-ipi - Posted interrupt processing Recognized interrupts are delivered without #VMEXIT - RVI/SVI are updated accordingly

18 APIC Virtualization and Virtual Interrupts 18 Posted Interrupt Processing Virtual interrupt are recorded in Posted Interrupt Descriptor Special Posted Interrupt Notification IPI - If, when the IPI is received, CPU is in a guest mode virtual interrupts from Posted Interrupt Descriptor are transfered to vapic page and processed by the CPU without #VMEXIT

19 APIC Virtualization and Virtual Interrupts 19 Result Eliminates up to 50% of #VMEXITs on I/O intensive workloads

20 Section 5 Nested VMX Improvements

21 Nested VMX Improvements 21 Problem: VMX Emulation is Expensive Each L1 VMREAD/VMWRITE is emulated by L0 There are many of them for each VMLAUNCH

22 Nested VMX Improvements 22 Problem: VMX Emulation is Expensive Each L1 VMREAD/VMWRITE is emulated by L0 There are many of them for each VMLAUNCH

23 Nested VMX Improvements 23 Solution: VMCS Shadowing L1 has shadow VMCS page linked from main VMCS L1 s VMREAD/VMWRITE reads/writes into shadow VMCS page without #VMEXIT When L1 executes VMLAUNCH L0 copies L2 VMCS values from the shadow VMCS page to VMCS02 (VMCS used by L0 to run L2) To minimize copying only most often used field are shadowed, access to the rest generates #VMEXIT as before

24 Nested VMX Improvements 24 Result Up to 40% less #VMEXITs

25 Nested VMX Improvements 25 Nested VMX ongoing development Nested EPT A lot of bug fixes lately

26 Section 6 MUMA Improvements

27 MUMA Improvements 27 Problem: Non Optimal Memory Placement

28 MUMA Improvements 28 Solution: Numa Aware Scheduler Memory follows CPU Periodically mark process memory as inaccessible Migrate memory to where the task is running now on NUMA fault Task follows memory Statistics of recent NUMA faults incurred on each node is kept per task Scheduler tries to run the task where its memory is

29 MUMA Improvements 29 Result 3-15% performance improvement.

30 Section 7 VDSO pvclock

31 VDSO pvclock 31 Problem Guest cannot use TSC directly as a clock source - May be unsynchronised between CPU sockets - Frequency may change due to migration Linux uses kvmclock as a clock source instead gettimeofday() does system call

32 VDSO pvclock 32 Solution: Add kvmclock Support to VDSO Map pvclock data structure into process memory Run kvmclock code from VDSO

33 VDSO pvclock 33 Result clock gettime() is reduced from 500 to 200 cycles

34 Section 8 Guest Spinlock Improvements

35 Guest Spinlock Improvements 35 Pause Loop Exit (PLE) optimisations Detect undercommit scenario: ebizzy (rec/sec higher is beter) before stdev after stdev %improve 1x x x dbench (throughput in MB/sec. higher is better) before stdev after stdev %improve 1x x x

36 Guest Spinlock Improvements 36 Pause Loop Exit (PLE) optimisations (Cont.) Detect preempted vcpus: ebizzy (rec/sec higher is beter) before stdev after stdev %improve 1x x x x kernbench (exec time in sec lower is beter) before stdev after stdev %improve 1x x x x

37 Guest Spinlock Improvements 37 Problem: Lock Waiter Preemption

38 Guest Spinlock Improvements 38 Problem: Lock Waiter Preemption

39 Guest Spinlock Improvements 39 Solution: Lock Waiter Preemption Work in Progress By Jiannan Ouyang 1 Preemptable Ticket Spinlock Downgrade a fair lock to an unfair lock automatically upon preemption, preserving the fairness otherwise. 1 preemptable_lock-ouyang-vee13.pdf

40 Section 9 More of Hyper-V Emulation

41 More of Hyper-V Emulation 41 Relaxes timing Disables Windows watchdog.

42 More of Hyper-V Emulation 42 Hyper-V Timers Reference Time Counter Per-partition reference time counter. Successive accesses return strictly monotonically increasing time values as determined by any and all virtual processors of a partition. It rate constant and unaffected by processor or bus-speed transitions or deep power-savings states. Partition Reference Time A reference time source that does not require an intercept into the hypervisor. This enlightenment is available only when the underlying platform provides support of an invariant processor TSC.

43 Section 10 PCI Device Assignment

44 PCI Device Assignment 44 PCI Device Assignment Improvements Virtual Function IO based KVM PCI device VFIO is a new UIO-like kernel driver that allows for a cleaner PCI device assignment architecture Move to this model as the primary device assignment mechanism More maintainable, better architecture, more secure interface for enabling PCI device assignment Legacy PCI device assignment is deprecated and now can be compiled out via kernel configuration

45 Section 11 Live Migration Updates

46 Live Migration Updates 46 Live Migration Improvements Better support for big guests xbzrle compression Migration thread Accurate migration downtime calculation

47 Live Migration Updates 47 Post-Copy Live Migration Latest patches use special device to trap guest memory access on a destination - Swapping, THP, KSM, NUMA balancing works only on anonymous memory Proposed solution: MADV USERFAULT & remap anon pages() - Guest memory VMAs marked with MADV USERFAULT - Guest s access to unmapped page causes special notification to be delivered to QEMU - QEMU receives missing page from migration to a local buffer and remaps it into the guest memory with remap anon pages() - Guest memory stays anonimous

48 Section 12 Block

49 Block 49 Block Live storage migration merged - Migration without shared storage GlusterFS block driver merged - Together with upcoming Gluster 3.4 release allows QEMU to bypass FUSE for increased performance Microsoft Hyper-V VHDX format support currently under development - To ease migration to KVM vhost-scsi - Use LIO (linux-iscsi.org) in kernel SCSI target code to handle SCSI protocol. - No QEMU userspace handling, no QEMU global mutex on data path - Code path is shorter, guest talks to host kernel directly - Higher performance (240K IOPS vs 12K IOPS for virtio-scsi)

50 Section 13 Networking

51 Networking 51 Networking Multi-queue NIC through virtio-net - Improves network performance and throughput for SMP guests - Incoming traffic to a guest scales linearly - Outgoing - sometimes a regression since more queues means more exits Bridge zero copy transmit - Zero copy transmit from guest to external traffic using network bridge - Improves network transmit performance for large message sizes - About 15% gain in CPU utilization

52 Section 14 virtio RNG

53 virtio RNG 53 Virtio Random Number Generator (RNG) Prevent entropy starvation in guests Inject entropy from host to the guest - The default mode uses the host s /dev/random - HW RNG device or EGD (Entropy Gathering Daemon) source

54 Section 15 New Hardware Architectures

55 New Hardware Architectures 55 New hardware Architectures ARM32 is merged ARM64 is on the way MIPS32 trap and emulate is merged MIPS-VZ is being worked on

56 Section 16 QEMU Consolidation

57 QEMU Consolidation 57 QEMU Consolidation qemu-kvm is no more

58 The end. Thanks for listening.

Red Hat Enterprise Virtualization Hypervisor Roadmap. Bhavna Sarathy Senior Technology Product Manager, Red Hat

Red Hat Enterprise Virtualization Hypervisor Roadmap Bhavna Sarathy Senior Technology Product Manager, Red Hat RHEV Hypervisor 1 RHEV Hypervisor Themes & 2 Architecture & Use cases 3 Q&A 4 Future 5 } HYPERVISOR