Supporting Soft Real-Time Tasks in the Xen Hypervisor

Transcription

1 Supporting Soft Real-Time Tasks in the Xen Hypervisor Min Lee 1, A. S. Krishnakumar 2, P. Krishnan 2, Navjot Singh 2, Shalini Yajnik 2 1 Georgia Institute of Technology Atlanta, GA, USA minlee@cc.gatech.edu 2 Avaya Labs Basking Ridge, NJ, USA {ask, pk, singh, shalini}@avaya.com The results described in this paper are elaborated upon in: Min Lee, A. S. Krishnakumar, P. Krishnan, Navjot Singh and Shalini Yajnik, Supporting Soft Real-Time Tasks in the Xen Hypervisor in Proceedings of the 2010 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, Pittsburgh, PA, March This document presents some of the key results of the research. ABSTRACT Virtualization technology enables server consolidation and has given an impetus to low-cost green data centers. However, current hypervisors do not provide adequate support for real-time applications, and this has limited the adoption of virtualization in some domains. Soft real-time applications, such as media-based ones, are impeded by components of virtualization including low-performance virtualization I/O, increased scheduling latency, and shared-cache contention. The virtual machine scheduler is central to all these issues. The goal in this paper is to adapt the virtual machine scheduler to be more soft-real-time friendly. We improve two aspects of the VMM scheduler managing scheduling latency as a first-class resource and managing shared caches. We use enterprise IP telephony as an illustrative soft real-time workload and design a scheduler S that incorporates the knowledge of soft realtime applications in all aspects of the scheduler to support responsiveness. For this we first define a laxity value that can be interpreted as the target scheduling latency that the workload desires. The load balancer is also designed to minimize the latency for real-time tasks. For cache management, we take cache-affinity into account for real time tasks and load-balance accordingly to prevent cache thrashing. We measured cache misses and demonstrated that cache management is essential for soft real time tasks. Although our scheduler S employs a different design philosophy, interestingly enough it can be implemented with simple modifications to the Xen hypervisor s credit scheduler. Our experiments demonstrate that the Xen scheduler with our modifications can support soft real-time guests well, without penalizing non-real-time domains. Keywords Virtualization, Xen, Enterprise telephony workloads, Server consolidation, Laxity * * This work was done when Min Lee was an intern at Avaya Labs.

2 Overview Real time and Xen Default Credit Scheduler Experiments Instrumentation Laxity-based scheduler Laxity (A) Boost with event (B) Simple Load Balance (L) Cache-aware Real-time load balancing (LL) Result Conclusion 1 Real time and Xen Near real-time applications pose challenges low-performance virtualization I/O scheduling latency shared-cache contention Scheduler is central to all these issues scheduling latency as a first-class resource managing shared caches 2 1

3 Deployment of Generic Enterprise Telephony system. Call-C and Sip-S (Call setup/tear down) IP Endpoint Gateway/ Media Server (stream encode/decode). Local Area Network Backbone Network. Local Area Network Gateway/ Media Server Dom0 Call-C Media-S Sip-S IP Endpoint Xen Hypervisor Hardware 3 Default Credit Scheduler Consumes credits to run VCPU4 VCPU2 VCPU7 VCPU1 V VCPU5 Under Over Every 30ms, distribute credits based on weights E.g. 20% CPU to VM 1, 40% CPU to VM 2 Proportional distribution Default weight (256) 4 2

4 Default Credit Scheduler Boost up waking-up tasks VCPU4 VCPU2 VCPU7 VCPU1 V VCPU5 Under Over New incoming (waking-up) task Probably due to external event (packet arrival) E.g. Domain 0 VCPU3 5 Default Credit Scheduler Boost up waking-up tasks VCPU3 VCPU4 VCPU2 VCPU7 VCPU1 V VCPU5 Boost Under Over VCPU4 VCPU2 VCPU7 VCPU1 VCPU3 V VCPU5 Only one time boost-up (<10ms) 6 3

5 Default Credit Scheduler Insufficient support for real-time tasks RT-tasks (Media-S) competes with other tasks Treated as just CPU-bound task Missing its deadlines 7 Credit Crisis Credit scheduler Good for CPU task Boost Boost waking-up task Short time (<10ms) Good for Dom0! Bad for Media task Low latency Need both CPU & Low latency! Competing with others System tasks Interactive tasks (i.e. Domain 0, supported by boost ) Multimedia tasks (Soft real time) CPU-bound tasks Background tasks (supported by credit) CPU allocation (credit) 8 4

6 Credit Crisis Now on, Laxity-based scheduler is introduced. Incremental (4 components) Experiments Instrumentation Laxity-based scheduler Laxity (A) Boost with event (B) Simple Load Balance (L) Cache-aware Real-time load balancing (LL) 9 Experiments (Deployment) Enterprise Telephony system Media server Media-S Signaling server Call-C,Sip-S Other VMs Dom0 Cdom [Computational domain] Two more domains [Licensing, management] Two flavors Standard (Cdom is empty) Cpuload (Cdom has 4 cpu-bound tasks) PESQ Quality of voice Major metric 10 5

7 COMP ACT Experiments (Setup) Dell 2950 server Dom0 Media-S SIP-S Call-C C-Dom 2 quad-core Xeon processors 4 GB of RAM 2 cores from each socket Xen Hypervisor Server So private cache 4cps, sample one out of 4calls G.711, 20ms packetization 30sec hold time IP IP network Max 240 streams through Media-S PESQ Compare the reference with stream from Media-S SIPp and RTP Clients tcpdump 11 Instrumentation Custom events for xentrace CPU utilization Worst/average wait time Each Time slice length Each Cache misses By xenoprof Average wait time L2 cache misses 12 6

8 Reading the plots PESQ Quality of voice 0(bad)~ (toll quality) Cumulative Density Boxplots Min/Max 25%,75% percentile Median Poor quality Good quality 13 Performance with default credit scheduler More weights (512, 1024) didn t improve much Only configuration it worked pinning Media-S/Call-C requires significant CPU Pinned Media-S/Call-C to,1 respectively Pinned others to CPU2,3 Dom0 is floating Underutilizing CPU! 14 7

9 Laxity (A) Kind of? Laxity! Target scheduling latency or deadline E.g. less than 5ms wait time in the run queue Parameter specified by user Real-time task Has laxity value Non-real-time task Doesn t have laxity value (or conceptually infinite laxity) 15 Implementation of laxity Where to insert real-time tasks to meet their deadline In over, laxity value is ignored. VCPU4 (1us) VCPU2 (7us) VCPU7 (4us) VCPU1 (13us) V (40us) VCPU5 (21us) 5us 20us 20us&boost Boost Under Over VCPU# (Length of previous time slice) 16 8

10 Prediction of wait time Each VCPU maintains an expected run time Previous time slice length The amount of CPU time it utilized in its previous run Works reasonably well Min/max 25%,75% percentile More sophisticated formula can be used Average difference between expected and actual wait times 17 Laxity Result Improved PESQ 18 9

11 Boost with event (B) Want to boost-up tasks receiving external event. Packet arrival. But credit scheduler only boost up waking-up task! Media-S in run queue doesn t get boosted So, we boost it up in the run queue VCPU4 (1us) VCPU2 (7us) VCPU7 (4us) VCPU1 (13us) V (40us) VCPU5 (21us) (2) Get boosted within queue (1) Receiving event VCPU4 (1us) VCPU1 (13us) VCPU2 (7us) VCPU7 (4us) V (40us) VCPU5 (21us) 19 Simple Load Balance (L) Default Credit scheduler VCPU4 VCPU2 VCPU7 VCPU1 V VCPU5 (2) Peek peer s Q (3) Steal higher task (1) Over or idle task? CPU1 VCPU10 VCPU

12 Simple Load Balance (L) Laxity-based scheduler (Simple Load Balance) VCPU4 VCPU2 VCPU7 VCPU1 V VCPU5 (2) Peek peer s Q (3) Steal higher task (4) If same (a) If mine is real time task, don t steal (b) If peer s is real time task, steal (c) Both non-real-time, compare entrance time into queue - with some delay (2ms) (1) Under, over or idle task? CPU1 VCPU17 VCPU11 VCPU10 VCPU15 21 Simple Load Balance (L) Laxity-based scheduler (Simple Load Balance) Media-S VCPU2 VCPU7 VCPU1 V VCPU5 (1) This effectively distributes real-time tasks over CPUs (2) Also prevents starvation CPU1 VCPU17 VCPU11 VCPU10 VCPU

13 Simple Load Balance (L) Laxity-based scheduler (Simple Load Balance) Media-S VCPU2 VCPU7 VCPU1 V VCPU5 (1) Kicks out non-real-time task if they waited for some time CPU1 VCPU2 VCPU17 VCPU11 VCPU10 VCPU15 23 Cache-aware Real-time load balancing (LL) Good, but Ping-ponging tasks trashes cache Bind RT-tasks to CPU Fix RT-tasks to its initial CPU Disable load-balancing for RT-tasks Unbalance of multiple RT-tasks Need new load-balancer for RT-tasks 24 12

14 Cache-aware Real-time load balancing (LL) Don t steal peer s real-time tasks in L Prefer hot cache (to low wait time) New x-sec-load balancer Balance of real time task s cpu utilization via bin-packing 90% utilized by one real time task CPU1 80% utilized by three real time tasks 25 Cache-aware Real-time load balancing (LL) Improved PESQ 26 13

15 Result (CPU utilization) We re fully utilizing CPUs! Amount of work done by cpuhogging process baseline(pinned) baseline A AL ALL Policies 27 Result (Various laxity value) Lower laxity more realtime better quality various laxity values - standard configuration 28 14

16 Result (Various laxity value) Lower laxity more realtime better quality various laxity values - cpuload configuration 29 Result (Boost with event) Some impact Adding Boost with event 30 15

17 Result (Load balance) Load balancing is essential for multicore PESQ improvement through load balancer in standard configuration 31 Result (Cache misses) We should consider cache Misses (x10000) baseline A AL ALL Total misses baseline A AL ALL baseline A AL ALL baseline A AL ALL baseline A AL ALL baseline A AL ALL C-dom Domain-0 Call-C Sip-S Media-S Cache misses for standard configuration 32 16

18 Conclusion New soft real time aware scheduler for Xen Better support for real time applications without penalizing non-real time tasks Fully utilizing CPU resources Timeliness requirement expressed by laxity Instrumentation to help design and measure performance Can also be used for other purposes 33 17