Tackling the Management Challenges of Server Consolidation on Multi-core System

Transcription

1 Tackling the Management Challenges of Server Consolidation on Multi-core System Hui Lv Intel June

2 Agenda SPECvirt_sc2010* Introduction SPECvirt_sc2010* Workload Scalability Analysis Hypervisor Overhead Analysis Credit Scheduler Optimizations Conclusions * The benchmark runs discussed here are for our research and non-compliant with the SPEC run-rules. The data presented here are only to illustrate the points discussed in this paper and cannot be compared with any other SPECvirt_sc2010 results 2

3 SPECvirt_sc2010* Workload Introduction Three sub-workloads: SPECjAppServer*, SPECimap*, SPECweb* Six VMs comprise a Tile to run as many as possible tiles Score: calculate arithmetic mean of the 3 normalized values per tile and sum the scores for all Tiles Infrastructure VM Webserver VM IMAP Server VM App Server VM Database VM Idle Server VM Tile 1 Virtualization Layer (XEN) and Hardware SPECweb2005* Driver SPECimap2007* Driver SPECjAppServer2004* Driver 3

4 Performance Scalability* Overview Performance scaling got worse as system load increased Response time became longer worse Qos * * Response time: Geomean of three kinds of sub-workload s response time * The benchmark runs discussed here are for our research and non-compliant with the SPEC run-rules. The data presented here are only to illustrate the points discussed in this paper and cannot be compared with any other SPECvirt_sc2010 results 4

5 CPU Cycles Components Breakdown Hypervisor occupied 28% of the total CPU cycles per transaction much high overheads! 5

6 Hypervisor Overhead Analysis The VMExit event of Ext Interrupt consumed ~48% of hypervisor cycles Context Switch consumed 27% of total hypervisor Cycles Most of the context switch happened in the VMExit event of External Interrupt Context switches: ~15k per second for one physical core at peak performance -- the average running tile slice for a vcpu once scheduled is less than 0.1 ms. * The cost of VMExit is calculated by removing domain0, cpuidle (7fff). It s the real overhead for hypervisor to process VMExit. * Context Switch means the process of de-schedule the current running vcpu and schedule in the next vcpu 6

7 Optimizations for Scheduler The process of scheduling consumed a big part in hypervisor. Meanwhile, high frequent context switch will also make cache cold thus increase the cycles per instructiion We worked out one way to optimize the scheduling process, so as to reduce overhead and improve performance 7

8 Generic Scheduler Process Pick up next vcpu Xen supplied generic API for specific implementation (credit1 and credit2) Two major parts in this flaw 1. To pick up next vcpu (SCHED_P) 2. To do context switch when selecting a new vcpu (SCHED_C) 8

9 Context Switch Rate Controller (SRC) Solution: To control scheduling rate in the following conditions 1) To skip the current scheduling process, if the frequency of context switch is bigger than the threshold during last period (10 ms) and last running vcpu is still runnable (not blocked) 2) To skip the current scheduling process, if last running vcpu runs less than some time slice (1ms) and still runnable Schedule Triggered Y Y VCPU1? Rate Control? VCPU1 Runnable Ret VCPU1 N? Running less than 1ms N Y N do_schedule 9

10 Performance Increase with SRC Optimization Perf/(cpu utilization) boosted by 15% Number of context switch reduced by 50%, thus cycles of hypervisor reduced by 22% Due to less context switch, decreased cache lower CPI lower CPU cycles for both Guest and Hypervisor Base With SRC SRC/Base Perf/(cpu cycles) 945 1, CPU% (Total) 92.00% 80.88% 0.88 Guest U 31.21% 28.56% 0.92 Guest K 31.58% 28.63% 0.91 Dom0 2.96% 3.20% 1.08 Xen 26.23% 20.48% 0.78 SCHED_Total 7.28% 4.40% 0.60 SCHED_Pick (credit) 2.40% 1.54% 0.64 SCHED_Context_Switch 2.33% 1.16% 0.50 Sched: runs through scheduler 6,312,866 5,304, Sched: context switches 6,008,568 3,329,

11 Credit1 vs. Credit2 Credit2 is the prototype brought in XEN 4.x. So far, it can work in complex consolidation environment Currently, overhead of credit2 is a bit higher than credit1 -- much faster pickup process in credit2, but slower context switch process Credit1 Credit2 Credit2/Credit1 Perf/transaction 1,254 1, CPU% (Total) 46.68% 54.47% 1.17 Guest U 15.21% 16.64% 1.09 Guest K 15.61% 17.24% 1.10 Dom0 1.82% 2.02% 1.11 Xen 14.04% 18.58% 1.32 SCHED_Total (cycles) SCHED_P (cycles) 1.32% 0.62% 0.47 SCHED_C (cycles) 0.95% 1.92% 2.02 Sched: runs through scheduler 6,339,737 5,808, Sched: context switches 4,689,289 4,615,

12 Conclusion Performance scalability got worse as system load increased in consolidation environment. Hypervisor composed a big part of the total system cycles, ~28% Too frequent context switch resulted in high overhead Some kind of rate controller for Credit scheduler benefit performance improvement Call people attention to continue developing a more powerful scheduler for Xen, in complex consolidation environment Intel and Xeon are trademarks of Intel Corporation in the United States and other countries * Other names and brands may be claimed as the property of others. 12

13 Backup 13

14 Hardware Layout SUT 3.33GHz Switch SR-IOV VFs iscsi Direct Link iscsi Target Clients Clients Storage Bay Intel Gbit Ethnet Adapter HBA Card 14

15 Server Under Test Configurations Processor Intel Xeon 5680 Sokets/Cores/Threads 2/12/24 Frequency 3.33GHz LLC 12MB BIOS HT ON, Turbo OFF, Power OFF, NUMA ON Memory 12 x 8GB DDR3 Platform S5520UR Controller LSI 3801 HBA Storage ISCSI for data disk, QEMU disk for OS disk Network G NIC Hypervisor Xen upstream c/s VM configs HVM Guests 15

16 Which Caused the Worse Scalability Cycles/transaction increase was caused by both CPI and Path Length increase -- Increase of CPI was partially due to increasing cache miss rate -- Increase of PL indicated some software bottlenecks existing 16

17 Hypervisor Events Overview Do we really need so many context switch work ~15k per second for one physical core at peak performance? It means the average running time slice for a vcpu once scheduled is less than 0.1 ms. Events (number/s) 1tile 9tile 9tile/1tile VMExits 55, , Hypercalls 52, , APIC timer interrupts 5,733 31, IRQ 10, , IPI 14, , sched: runs through schedule 42, , sched: context switches 28, , csched: migrate_queued 7 39,757 5,847 csched: migrate_running 0 3 N/A 17

18 VMExit Events Distribution At peak performance, top three VMExit events were APIC Access, External Interrupt and CR Access However, larger number does not mean higher overhead it depends on the cost of related VMExit event 18