CFS-v: I/O Demand-driven VM Scheduler in KVM

Transcription

1 CFS-v: Demand-driven VM Scheduler in KVM Hyotaek Shim and Sung-Min Lee (hyotaek.shim, Software R&D Center, Samsung Electronics

2 Problem in Server Consolidation 2/16 bound VMs are often co-located with computing bound VMs on the same physical machine To avoid significant performance interference from computing-andcomputing VMs or -and- VMs Comp. Comp. Comp. Comp. Guest Kernel Guest Kernel Guest Kernel Guest Kernel KVM KVM Physical Machine Physical Machine TRACON: Interference-Aware Scheduling for Data-Intensive Applications in Virtualized Environment (SC 11) Understanding Performance Interference of Workload in Virtualized Cloud Environments (CLOUD 10)

3 Problem in Server Consolidation 3/16 performance is still interfered with co-running computing bound VMs Also when computing and tasks coexist on the same VM Performance Degradation Performance Degradation Performance Degradation Performance Degradation Comp. Comp. Comp. Comp. Guest Kernel Guest Kernel Guest Kernel Guest Kernel KVM KVM Physical Machine Physical Machine

4 Performance Evaluation of CFS in KVM 4/16 performance interference by computing tasks Mixed (4 VMs): four VMs, each of which has 8 tasks and 1~4 comp. tasks Separate (2 VMs): VM1 (32 tasks), VM2 (16 Comp. tasks) No Interference without computing tasks No Degradation at all regardless of computing threads IOPS (4KB Random Read) VM (Mixed) VM (Mixed, 4C) VM (Mixed, 8C) VM (Mixed, 16C) VM (Separate) Native Linux Native Linux (4C) Native Linux (8C) Native Linux (16C)

5 Performance Evaluation of CFS in KVM 5/16 CPU utilization of VMs(Mixed) and Native Linux Ratio (%) 100% 80% 60% 40% Without C-Threads idle iowait guest+steal sys+irq+soft user+nice Ratio (%) 100% 80% 60% 40% With 4 C-Threads idle iowait guest+steal sys+irq+soft user+nice 20% 20% 0% Host VM1 VM2 VM3 VM4 Linux 0% Host VM1 VM2 VM3 VM4 Linux Ratio (%) 100% 80% 60% 40% With 8 C-Threads idle iowait guest+steal sys+irq+soft user+nice Ratio (%) 100% 80% 60% 40% With 16 C-Threads idle iowait guest+steal sys+irq+soft user+nice 20% 20% 0% Host VM1 VM2 VM3 VM4 Linux 0% Host VM1 VM2 VM3 VM4 Linux

6 CFS in Native Linux 6/16 In native Linux CFS, -bound tasks could repeatedly preempt computing-bound tasks while consuming a small time slice Wakeup & Preemption Wakeup & Preemption Wakeup & Preemption Wakeup & Preemption I O Comp. I O Comp. I O Comp. I O Comp. I O Comp. What if an bound task repeatedly fail to preempt the current task? Wakeup & Fail to preempt Wakeup & Fail to preempt I O Comp. I O Comp. Time Line

7 Path of Virtio-blk 7/16 Five different kinds of tasks are involved for handling a virtio-blk request Each task may be woken up just before handling their job (totally, up to five wake-ups) Main Loop Thread Bdrv Request Worker Thread Vring Interrupt VCPU0 Thread RESCHEDULE IPI VCPU1 Thread Event Interrupt (Intr.+Softirq) Idle Request PCPU0 PCPU1 PCPU2 PCPU3 Host Kernel Softirq Event (ioeventfd)

8 Problem of CFS on VM scheduling 8/16 In CFS group scheduling, VMs are deployed in different groups (runqueues) Group entity of woken-up related threads often fail to preempt the group entity of computing VCPUs CFS Runqueue Woken-up but fails to preempt Sched Entity Sched Entity CFS Runqueue CFS Runqueue Event Interrupt Request Event Computing Computing Main Loop Thread Worker Thread VCPU0 VCPU1 Current VCPU0 VCPU1

9 Preemption Rule of CFS 9/16 Vruntime Compensation When a schedule entity is woken up, the entity s Vruntime is reset to minimum Vruntime in its runqueue - threshold(e.g., ) The reset Vruntime is not small enough to preempt the current computing task Weight and Grain Schedule entity for a group of bound VM tasks has a relatively small Weight due to CPU consumption, which results in large Grain If woken-up sched entity s Vruntime is smaller than Current sched entity s Vruntime + Grain, then preempt the current task! Grain is calculated by Weight of the woken-up

10 CFS-v: Boosting Related Tasks 10/16 CFS-v detects bound VCPU tasks by using a 2-bit count If a woken-up task consumes CPU less than 500us, increases by 1 If it consumes time slice more than 1ms, the count decreases by 1 The count is larger than or equal to 2, the VCPU is related Main loop, worker, and softirq threads are considered related A woken-up related task can always preempt non related tasks Next task Boost Queue CFS Runqueue CFS Runqueue Event Interrupt Request Computing Computing Computing Computing Main Loop Thread Worker Thread VCPU0 VCPU1 VCPU0 VCPU1 VCPU0 VCPU1

11 CFS-v: Partial Boosting 11/16 VCPU that has both and computing tasks is not considered as bound and thus cannot be boosted with the previous boosting algorithm What if CFS-v boosts such a VCPU? The VCPU will not return CPU shortly due to computing tasks bound Task Current task in VCPU0 CFS Runqueue Performance Degradation C Computing bound Task Runqueue Runqueue Wait (in the wait queue) Running Comp. Ready but not running CFS Scheduler Guest VM Kernel CFS Scheduler Guest VM Kernel VCPU0 VCPU1

12 CFS-v: Partial Boosting 12/16 Even for a non related VCPU, if it receives Vring Interrupt or RESCHEDULE IPI, it is partially boosted within 200us time slice After a timer interrupt, the partially-boosted VCPU returns CPU Timer (200us) CFS Runqueue Sched Entity Sched Entity Sched Entity Com Com Com Com Com VCPU Task VCPU Task VCPU Task VCPU Task Boost Queue (Partial Boosting in Xen) Task-aware Virtual Machine Scheduling for IO Performance (VEE '09)

13 Performance Evaluation 13/16 Evaluation Environment Host: Intel(R) Core(TM) i CPU 3.40GHz (8 Cores), 12GB DRAM, kernel Guest VMs: kernel , 8 Cores, 512MB DRAM» Running on a QCOW2 image, sharing a hard disk» Storage benchmark on RAW, sharing a Samsung 256GB SSD 840pro 2.1, Virtio-blk We modified the host kernel and to implement CFS-v Micro benchmark Storage benchmark 4KB O_DIRECT mode random read Host kernel s page cache is also disabled Computing benchmark MD5 Hashing

14 Performance Evaluation 14/16 Separate 2VMs: co-running 2 VMs that execute and computing workloads, respectively VM1: 32 random read threads VM2: 16 MD5 hashing threads CFS-v improves the storage performance of VM1 by 764%, while reduces the computing performance of VM2 by 30.1% CFS-v has also raised the CPU util. of VM1 from 46% to 273%

15 Performance Evaluation 15/16 Mixed 2VMs: co-running 2 VMs, each of which is running 16 random read threads and 8 MD5 hashing threads CFS-v improves the storage performance by 93% and reduces the computing performance by only 9%

16 Summary 16/16 CFS-v enhances the overall resource efficiency of server consolidation Enables KVM to detect related and system tasks and to boost them ahead of computing tasks By using partial boosting, CFS-v can also timely and momentary boost bound tasks inside a VCPU running mixed workloads CFS-v highly improves the performance of VMs in exchange for little degradation on the computing performance