Correlation-Aware Virtual Machine Allocation for Energy-Efficient Datacenters

Transcription

1 Correlaton-Aware Vrtual Machne Allocaton for Energy-Effcent Datacenters Jungsoo Km Martno Ruggero Davd Atenza Embedded Systems Lab (ESL), EPFL Emal: Marcel Lederberger Datacenter Faclty Management, Credt Susse AG Emal: Abstract Server consoldaton plays a key role to mtgate the contnuous power ncrease of datacenters. The recent advent of scale-out applcatons (e.g., web search, MapReduce, etc.) necesstate the revst of exstng server consoldaton solutons due to dstnctvely dfferent characterstcs compared to tradtonal hgh-performance computng (HPC),.e., user nteractve, latency crtcal, and operatons on large data sets splt across a number of servers. Ths paper presents a power savng soluton for datacenters that especally targets the dstnctve characterstcs of the scale-out applcatons. More specfcally, we take nto account correlaton nformaton of core utlzaton among vrtual machnes (VMs) n server consoldaton to lower actual peak server utlzaton. Then, we utlze ths reducton to acheve further power savngs by aggressvely-yet-safely lowerng the server operatng voltage and frequency level. We have valdated the effectveness of the proposed soluton usng 1) multple clusters of real-lfe scale-out applcaton workloads based web search and 2) utlzaton traces obtaned from real datacenter setups. Accordng to our experments, the proposed soluton provdes up to 13.7% power savngs wth up to 15.6% mprovement of Qualty-of-Servce (QoS) compared to exstng correlaton-aware VM allocaton schemes for datacenters. I. INTRODUCTION The soarng demand for computng has produced as collateral undesrable effect a surge n power consumpton of servers and datacenters [1]. Server consoldaton [3], whch mnmzes the number of actve servers by packng workloads, or vrtual machnes (VMs) n a vrtualzed envronment, nto the mnmal number of actve servers, s one of the wdely used technques to reduce the power consumpton of datacenters. However, n order to satsfy the performance demand of applcatons runnng on servers, server consoldaton s usually conducted assumng the worst-case (or peak) utlzaton [2], [3]. Thus, n order to acheve power consumpton wthout any sgnfcant Qualty-of-Servce (QoS) degradaton, many works have presented aggressve consoldaton schemes, whch pack VMs based on off-peak (e.g., 90th/95th/99th percentle) of server utlzaton [4], [5]. Recently, correlaton of resources utlzaton patterns among VMs are also exploted, such that, un-correlated VMs are co-located nto a server to enable overprovson of VMs under neglgble QoS degradaton [6] [9]. Nonetheless, these exstng solutons are mostly desgned for hgh-performance computng (HPC) applcatons and do not Ths work descrbed n ths paper has been partally supported by the PMSM: CT Montorng research grant for ESL-EPFL funded by Credt Susse AG and an ERO Research Grant from Oracle for ESL-EPFL. work well for emergng cloud (or scale-out [10]) applcatons (e.g., web search, MapReduce, etc.) due to the lack of consderatons of the characterstcs of the scale-out applcatons. In partcular, the characterstcs of scale-out applcatons are qute dfferent from tradtonal HPC workloads n both macroscopc and mcroscopc scales. At the macroscopc scale, the applcaton, frst, s user-nteractve; thereby, the amount of requred computng capacty s hghly varable and fastchangng [5] due to the dependence wth external factors, e.g., number of clents/queres, etc. Second, the responsveness (or latency) should come at the frst crtera to be satsfed as the level of user satsfacton leads to the success of the busness [11]. Fnally, the amounts of requred CPU and memory resources are usually far beyond the level that a sngle server can sustan. Hence, massvely parallel nodes are cooperatvely workng by formng a cluster archtecture [12]. For nstance, n a web search applcaton, a bg set of search ndexes s dvded nto multple smaller datasets, and then allocated nto multple VMs (or servers), each of whch s called an ndex searchng node (ISN). Once a query s arrved, each ISN ndependently searches matched data wth the allocated dataset and a master (.e., front-end) node gathers the search results from the multple ISNs, then sends the results to clents. At the mcroscopc-scale, the characterstcs of scaleout applcatons are well studed n [10]. Among varous characterstcs, the memory footprnt s far beyond the amount an on-chp cache can sustan; thereby, ncreasng the on-chp cache sze only produces neglgble performance mprovement. Because of these aforementoned dscrepances wth HPC workloads, exstng datacenter power management solutons, whch neglect or only partally consder the characterstcs of scale-out applcatons, do not explot all the opportuntes to acheve global power savngs. In ths paper, we propose a dynamc power management soluton for servers hostng these new scale-out applcatons, especally accountng for the correlaton nformaton among VMs, whle satsfyng peak resource requrements. Compared to exstng correlaton-aware solutons, the contrbutons of ths work are as follows: We analyze workload characterstcs of scale-out applcatons and present new opportuntes for power management n vrtualzed server envronments. We present a novel power management soluton jontly /DATE13/ c 2013 EDAA

2 utlzng server consoldaton and voltage and frequency (hereafter, v/f) scalng consderng the characterstcs of scale-out applcatons, especally correlaton among VMs. We valdate the applcablty of the proposed soluton wth the real deployment of multple dstrbuted web search applcatons taken from CloudSute [10]. We valdate the effectveness of the proposed soluton to larger scale problems usng the utlzaton traces obtaned from acutal datacenters, whch provdes up to 13.7% power savngs and 15.6% QoS mprovement compared to exstng correlaton-aware schemes [6]. Ths paper s organzed as follows. Secton II revews the related work on datacenter power management. Secton III analyzes the characterstcs of scale-out applcatons and revsts the power management approaches consderng the characterstcs. Secton IV proposes a novel VM allocaton and v/f scalng solutons. Secton V presents our expermental results, followed by conclusons n Secton VI. II. RELATED WORK Varous server consoldaton solutons are proposed based on per-vm workload characterstcs,.e., the peak utlzaton of VMs [2], [3] and off-peak (e.g., 90th/95th/99th percentle) values observng that the peak utlzaton happens rarely and t s much hgher (more than 2x) than 95th and 99th percentle values [4], [5]. To acheve further power savngs whle mantanng QoS level, jont relatonshps among VMs, lke correlatons, have been exploted n recent works [6] [9]. In [6], Verma et al. presented a clusterng-based correlatonaware VM placement soluton. The soluton frst clusters VMs, such that, the envelops of VMs CPU utlzaton (defned as a bnary sequence where the value becomes 1 when CPU utlzaton s hgher than the off-peak value, otherwse, 0 ) ncluded n dfferent clusters do not overlap. Then, t allocates VMs to servers n order to co-locate VMs n dfferent clusters by provsonng VMs based on ther off-peak utlzaton demands (e.g., 90th percentle), whle reservng a shared peak buffer to handle resource demand hgher than the off-peak value for all co-located VMs. However, ths approach s applcable only when the envelops of VMs are statonary and dstnctvely dfferent one from another, thereby, producng multple clusters. Hence, t does not work well wth scaleout applcatons wth non-statonary and fast-changng VM behavors. Then, n [7], Meng et al. proposed a jont-vm szng technque that pars two un-correlated VMs nto a super- VM and provson super-vms by predctng the the aggregated workloads. However, once super-vms are formed, ths soluton does not consder the correlatons of VMs wthn a same super- VM anymore. Thus, t may lose the chance of further power savngs by leveragng tme-varyng correlatons n scale-out applcatons. In [9], Halder et al. extends the scheme such that aggregated workload of multple VMs can be utlzed for VM placement. However, ths soluton can be applcable only when future servers utlzaton s perfectly known. In summary, all exstng solutons do not properly capture the characterstcs of scale-out applcatons. Thus, we need to develop a power management soluton for datacenters by accountng for ths specfc characterstcs to acheve sutable power savngs whle satsfyng performance requrements. III. NEW OPPORTUNITIES FOR POWER MANAGEMENT Power management solutons for datacenters hostng scaleout applcatons should be dfferent from the case of hostng HPC applcatons due to the dstnctve characterstcs of scaleout applcatons. In ths secton, we present three prncples of dynamc power management solutons for datacenters hostng the scale-out applcatons based on our observatons. All data presented n ths secton s measured usng an AMD Opteron 6174 archtecture wthn a DELL PowerEdge R815 server. A. Conservatve resource provson wth v/f scalng Scale-out applcatons are user-nteractve. Therefore, responsveness, n terms of latency, s the frst prorty to be met [11]. Moreover, every applcaton (or VM) s assumed to be equally mportant n clouds. Thus, we should conservatvely provson VMs based on the peak (or Nth percentle accordng to QoS requrement) resource demand of each VM. The requred QoS level can be acheved by assgnng the rght number of cores because the performance s hghly scalable to the number of allocated cores due to the hgh parallelsm of such applcatons. Moreover, the resource demand s tmevaryng and s mostly lower than the value used for the core allocaton. However, as descrbed n [5], dynamc power gatng (turnng on/off cores) cannot be applcable to such applcatons due to the sgnfcant performance degradaton caused by the long transton latency between power modes and fast changes of resource demands. Thus, dynamc v/f scalng s the only soluton to acheve power savngs whle satsfyng the performance requrement. Motvated by these observaton, the proposed soluton allocates the number of cores for each VM accordng to ts peak (or off-peak dependng on QoS level) resource demand to guarantee equal QoS levels to all VMs whle scalng v/f level to acheve power savngs. B. Sharng cores among co-located VMs The amount of requred CPU utlzaton vares as the amount of user requests to servers changes over tme. Fg. 1 shows the CPU utlzaton traces for two VMs, each of whch s an ndex servng node (ISN), n a sngle web search cluster to process queres requested from the varyng number of clents. As shown n the fgure, the CPU utlzatons of both VMs are hghly synchronzed wth the varaton of the number of clents. Furthermore, loads between VMs n a cluster are not perfectly balanced because the CPU utlzaton depends on the amount of matched results correspondng to a user request. Thus, we can mprove the resource utlzaton by sharng cores among multple VMs, such that they can flexbly use cores dependng on ther tme-varyng resource demands. Furthermore, as analyzed n [10], the overhead of sharng cores s neglgble due to the large memory footprnt of scale-out applcatons,.e., far beyond the capacty of on-chp caches. Table I shows the measured performance metrcs used of a web search applcaton when t s co-located wth varous applcatons (from PARSEC benchmark sute). We compared

3 VM 2 Number of clents Fg. 1. Varatons of CPU utlzaton of two ndex searchng nodes (ISNs) wth respect the number of clents TABLE I PERFORMANCE METRICS OF A WEB SEARCH APPLICATION CO-LOCATED WITH A VM RUNNING PARSEC BENCHMARK: NUMBERS IN PARENTHESIS SHOW THE CASE WHEN A WEB SEARCH APPLICATION IS RUNNING ALONE VM 1 IPC L2 MPKI L2 mss rate (%) w/ Backshcoles 0.76 (0.75) 2.38 (2.40) (11.57) w/ Swaptons 0.75 (0.77) 2.32 (2.43) (9.63) w/ Facesm 0.70 (0.70) 2.41 (2.36) (11.31) w/ Canneal 0.76 (0.78) 2.46 (2.43) (11.67) nstructon per clock cycles (IPC), L2 mss-per-klo-nstructon (MPKI), and L2 mss rato (%). The values are obtaned usng Xenoprof patched for the AMD15h Bulldozer archtecture [14]. The numbers n parenthess show the case before colocaton. As can be seen, there are only neglgble varatons over all the metrcs, whch correspond to a neglgble performance degradaton due to cores sharng. Motvated by these observatons, the proposed soluton allocates VMs to servers such that all co-located VMs share cores assumng that the performance degradaton s neglgble. C. Correlaton-aware VM placement Due to the dstrbuted operatons of multple VMs n a cluster, we can observe a hgh correlaton wthn a cluster of scale-out applcatons, called ntra-cluster correlaton, rather than the correlaton among dfferent clusters (or servces) targeted n other correlaton-aware scheme [6] [9]. In Fg. 1, we can observe the ntra-cluster correlaton between two VMs n a cluster, both of whch are strongly synchronzed wth the varaton of the number of clents. Thus, the proposed soluton takes nto account the pervasve correlaton n scaleout applcatons,.e., wthn a cluster as well as among clusters, such that correlated VMs are not co-located. IV. CORRELATION-AWARE POWER MANAGEMENT In ths secton, we present the proposed datacenter power management soluton based on the clams n the prevous secton. Frst, we defne a cost functon to effcently quantfy the level of correlaton used n the proposed VM placement (Secton IV-A). Second, we propose the correlaton-aware VM allocaton scheme (Secton IV-B) whle sharng cores among co-located VMs. Fnally, we provde a way to scale the v/f level to acheve power savngs wthout any QoS degradaton (Secton IV-C). Note that we assume that servers are homogeneous, and where each of them conssts of N core cores wth multple frequency levels. A. Effcent correlaton measure for VM allocaton The correlaton of used CPU utlzaton between two VMs s mostly quantfed wth Pearson product-moment correlaton coeffcent, or Pearson s correlaton [8], whch s calculated as the rato of covarance of the two random varables to the product of ther standard devatons. However, the overhead to calculate the metrc for a certan tme nterval s hgh for a short tme perod because the computaton s concentrated at the end of the tme perod, as t utlzes the average values of CPU utlzaton samples, whch are collected durng each tme perod. In addton, Pearson s correlaton s also partly neffcent because the value reflects correlaton throughout the correspondng tme nterval, even though we only requre the correlaton at (off-)peak utlzatons n VM placement. To overcome the drawback and neffcency n ths metrc, we propose a new cost functon to quantfy correlaton between two VMs (n terms of CPU utlzaton), as follows: Cost vm,j = ûcpu(v M ) + û cpu (V M j ) û cpu (V M + V M j ) (1) where Cost vm,j represents the newly defned correlaton measure between V M and V M j. û cpu (V M ) s a reference utlzaton of V M, whch s ether the peak or the Nth percentle value dependng on QoS requrement. The numerator represents the worst-case peak CPU utlzaton when the peaks of two VMs concde, whle the denomnator s an aggregated actual peak utlzaton when V M and V M j are colocated nto a same server. Thus, the hgher Cost vm,j, the lower correlaton between V M and V M j. Moreover, we can update the values at each samplng perod of utlzaton. Thus, we can save memory space to store all samples as well as evenly dstrbutng computatonal effort to measure the correlaton across a certan tme horzon. Usng our new Cost vm,j functon, we can model correlatons among all VMs by constructng a 2-D matrx, namely, Mcost vm where the (,j)-th element corresponds to Cost vm,j. B. Correlaton-aware VM allocaton We allocate VMs such that the correlaton among the allocated VMs n Server,.e., V alloc = {V M,1,, V M,n vm } where n vm s the number of VMs allocated to Server, s mnmzed, whle the sum of û cpu (V M,j ) n the server does not exceed the total CPU capablty of the server,.e., Cap, as well as the number of the actve servers s mnmzed. The correlaton of Server s defned as shown n Eqn. (2): Cost server = n vm j=1 w vm,j ( Nvm k=1&k j Cost vm j,k n vm 1 where w,j vm represents a weght of V M,j, defned as the rato of û(v M,j ) to the sum of û(v M,j ) s of all co-located VMs n Server. The problem of fndng optmal sets of VMs s a well-known bn-packng problem [15]. To reduce the soluton complexty, we propose a soluton based on a Frst-Ft-Decreasng heurstc as shown n Fg. 2. Our proposed algorthm s perodcally nvoked at every t perod. The algorthm s largely dvded nto two phases: 1) UPDATE (lnes 1 8) and 2) ALLOCATE (lnes 9 18). In the UPDATE phase, we ntalze parameters and update CPU utlzaton statstcs. Then, we allocate VMs to servers n the ALLOCATE phase. ) (2)

4 Fg. 2. The proposed correlaton-aware VM placement consstng of UPDATE and ALLOCATE phases In the UPDATE phase, we frst ntalze a set of unallocated VMs (V unalloc ), sets of allocated VMs (V alloc ), remanng capacty (Rem ) for all servers, and a correlaton threshold (T H cost ) n lnes 1 4. Second, we predct the workload based on hstory, as we prevously prepared n [15] (lne 5). Thrd, we sort VMs n V unalloc n descendng order of predcted û cpu (V M ) to reduce the fragmentaton of the bn-packng problem (lne 6). Fourth, we update Mcorr vm by updatng the Cost vm,j for all VM pars (lne 7). Fnally, we determne the number of estmated actve servers,.e., Ñ server, as presented n Eqn. (3) (n lne 8): Nvm =1 Ñ server = û cpu (V M ) (3) N core where û cpu represents an estmate of û cpu. Then, Ñ server s equal to the mnmum number of servers to accommodate all VMs n V unalloc. We provson VMs to reduce the number of actve servers whle satsfyng performance requrements. The ALLOCATE phase s terated untl all VMs are allocated to Ñserver servers (lne 9). Frst, we select a server havng the largest remanng CPU capablty,.e., Rem (lne 10). Second, we fnd a VM to be allocated nto Server (lne 11), whch has the hghest Cost server wth VMs n V alloc, whle satsfyng two condtons: 1) Cost server should be larger than T H cost ; and 2) û cpu (V M ) should be less than or equal to Rem. In case we fnd a VM, we update V alloc, Rem, and V unalloc accordngly (lnes 12 15). The procedure to fnd VMs to be allocated n Server s terated untl there s VM left (lnes 12 16). If we have unallocated VMs at the end of the teraton, we repeat the procedure (from lnes 10 16) wth a degenerated T H cost by a factor of α (lne 17) along wth a lst of servers sorted n descendng order of Rem (lne 18). C. Decson of v/f level Once all VMs are allocated nto servers, we determne an optmal v/f level for each server. However, we cannot exactly estmate how much we can lower v/f level when multple VMs are allocated n a server because Cost vm,j UPDATE ALLOCATE only captures Y=X Fg. 3. Relatonshp between weghted average correlaton n Eqn. (2) and possble v/f scalng factor: the lower bound of the possble v/f scalng factor has lnear relatonshp wth Cost server the correlaton between two VMs. Therefore, we emprcally calculate the lower bound of v/f slowdown through Cost server n Eqn. (2), as shown n Fg. 3. X- and Y-axes, respectvely, represent a weghted average cost functon calculated wth Eqn. (2) and the rato of the sum of û cpu (V M ) s of colocated VMs to the aggregated peak value of the server, whch represents possble v/f slowdown. Based on the relatonshp, we can determne the frequency level of Server,.e., f, as presented n Eqn. (4): ( ) ( n vm 1 j=1 f = Cost server ûcpu(v ) M,j ) N core server f max (4) where f max s the maxmum frequency level. f s set by lowerng the worst-case peak requred frequency level (.e., the second parenthess assumng the stuaton when peaks of VMs concde) wth a factor of 1/Cost server. V. EXPERIMENTAL RESULTS We valdated the proposed datacener power management approach n two setups. Frst, we appled the proposed soluton to two web search clusters runnng on DELL PowerEdge R815 servers to valdate the applcablty of the proposed correlatonaware scheme for scale-out applcatons. Second, we further nvestgated the effectveness to larger scale problems wth the utlzaton traces obtaned from a real datacenter setup. A. Setup-1: Dstrbuted web search applcatons We bult two web search clusters,.e., Cluster 1, and Cluster 2, usng the CloudSute benchmarks [10]. Each cluster conssts of three VMs: one s front-end (Tomcat ) and two are ISNs (Nutch-1.2). Note that the CPU utlzaton of the front-end s qute low compared to ISNs. Thus, we smply vared the allocaton of VMs hostng ISNs. We annotate four ISNs as V M 1,1, V M 1,2, V M 2,1, and V M 2,2 where {V M 1,1, V M 1,2 } and {V M 2,1, V M 2,2 } are ncluded n Cluster 1 and Cluster 2, respectvely. We used Xen-4.1 hypervsor for server vrtualzaton and each VM has Ubuntu11.10 as ts operatng system (OS). We emulated clents behavor usng Faban-0.7 and vared the number of clents from wth the form of sne and cosne waves for Cluster 1 and Cluster 2, respectvely. We used two servers each of whch conssts of 8 cores havng two frequency levels,.e., 1.9GHz and 2.1GHz. We compared three dfferent VM allocatons, as llustrated n the upper part of Fg. 4. 1) Segregated where

5 VM1,1 VM1,2 VM2,1 VM2,2 VM1,2 (8 cores) VM1,1 (8 cores) VM2,2 (8 cores) VM2,1 (8 cores) VM2,1 (8 cores) VM1,1 (8 cores) VM2,2 (8 cores) VM1,2 (8 cores) Server1 (8cores) Server2 (8cores) Server1 (8cores) Server2 (8cores) Server1 (8cores) Server2 (8cores) VM1,1 VM1,2 VM2,1 VM2,2 Server2 Server1 Server2 Server1 (a) Fg. 4. (b) VM placements and CPU utlzaton traces of (a) Isolated, (b) Shared-UnCorr, and (c) Shared-Corr each VM s ndependently runnng on 4 cores each, 2) Shared- UnCorr where 8 cores are shared wth two VMs n a same cluster (.e., correlaton unawareness), and 3) Shared-Corr where 8 cores are shared wth two VMs n dfferent clusters (.e., ncludng correlaton awareness). Then, Fg. 5 shows comparsons n terms of the 90th percentle response tme. As ths fgure ndcates, the 90th percentle response tme n Shared-UnCorr s lower than Segrated by 43.6% (from to sec) whle Shared- Corr provdes another 7.7% lower response tme (from to sec) than Shared-UnCorr under 2.1GHz. The results can be explaned by observng the CPU utlzaton traces n Fg. 4. The X- and Y-axes represent the elapsed tme (n sec) and the normalzed CPU utlzaton wth respect to the number of servers, respectvely. The samples are collected at every 1 sec usng a Perl scrpt montorng tool Xenstat.pl. The reason of the hgh response tme n Segregated case s the neffcent utlzaton of the allocated cores. As shown n Fg. 4(a), V M 1,1 and V M 2,2 are under-utlzed whle V M 1,2 and V M 2,1 are over-utlzed,.e., approachng ther maxmum CPU utlzaton levels, and needs more than 4 cores. Note that the response tme of the dstrbuted web search cluster s constraned by the latest VM because a front-end sends results to clents only after collectng the search results from all ISNs. Thus, due to the defcency of the CPU capablty of the over-utlzed VMs, queres must wat n a queue for a longer tme before beng processed. Thus, the response tme of Segregated case becomes longer. On the contrary, Shared-UnCorr enables to effcently use all the 8 cores n each server by flexbly schedulng VMs to the cores accordng to ther tme-varyng demands. Ths result supports our clam n Secton III-B where we antcpated that the gan attanng from sharng cores among VMs s much hgher than the performance degradaton caused by the nterference among co-located VMs. However, the maxmum CPU utlzaton reaches up to 0.88 because two VMs wthn the same cluster are hghly correlated. Hence, the peaks of the CPU utlzatons concde. Such hgh CPU utlzaton can result n longer response tmes [13]. We can reduce the peak utlzaton by allocatng VM consderng correlatons among VMs n Shared-Corr (Fg. 4(c)). In Shared-Corre, the maxmum CPU utlzaton becomes even and lowered down to 0.6. The mproved response tme n Shared-Corr can be 90th percentle Reponse tme (sec) Cluster1 (c) Cluster Segrated Shared UnCorr Shared Corr (2.1G) Shared Corr (1.9G) Fg th percentle response tme of Cluster 1 and Cluster 2 for three dfferent VM allocatons used to save power consumpton by lowerng the frequency level. As shown n Fg. 5, Shared-Corr runnng wth 1.9GHz provdes almost smlar response tme (0.155 vs sec) to Shared runnng wth 2.1GHz, whch results n approxmately 12% powe savngs. B. Setup-2: Utlzaton traces obtaned from datacenter setups To further nvestgate the effectveness of the proposed soluton, we performed another set of smulatons usng utlzaton traces obtaned from an actual datacenter. As most of VMs are severely under-utlzed, we selected the top 40 VMs n terms of CPU utlzaton. We sampled the CPU utlzaton every 5 mn. for a day whle syntheszng fne-graned samples per 5 sec. wth a lognormal random number generator [16], whose mean s the same as the collected value for the correspondng 5- mnute sample rate. Usng ths utlzaton traces, we evaluated the effectveness of the proposed soluton wth a vrtual testbed consstng of 20 servers. We targeted an Intel Xeon E5410 server confguraton whch conssts of 8 cores and two frequency levels (2.0GHz and 2.3GHz), and used the power model proposed n [13]. We performed VM placement every 1 hour,.e., t perod =1 hour, wth predctons of upcomng workloads usng a last-value predctor. Then, we compared the followng three approaches of power management for datacenters: Best-Ft-Decreasng (BFD): a conventonal best-ftdecreasng heurstc approach. Peak Clusterng-based Placement (PCP) [6]: a correlaton-aware VM allocaton whch clusters VMs usng ts Envelope-based correlaton classfcaton. Proposed: the proposed correlaton-aware VM allocaton. Table II(a) compares the power consumpton and performance volatons of the three approaches when we statcally

6 TABLE II COMPARISONS FOR (A) STATIC AND (B) DYNAMIC V/F SCALING (a) Normalzed power Maxmum volatons (%) BFD PCP [6] Proposed (b) Normalzed power Maxmum volatons (%) BFD PCP Proposed (a) (b) Fg. 6. Comparson of frequency dstrbutons n (a) Server 1 and (b) Server 3 set the v/f level at the tme of VM placement,.e., t perod. The power consumpton results are normalzed wth respect to the power consumed by BFD, and the maxmum volaton shows the maxmum per-perod rato of the number of over-utlzed tme nstances (.e., when the aggregated utlzaton among colocated VMs s beyond the CPU capacty of a correspondng server) to t perod, durng the entre perods,.e., 24 hours. The proposed soluton provdes up to 13.7% power savngs compared to BFD and PCP, whle drastcally reducng the number of the volatons. It s noteworthy that PCP provdes almost smlar results wth BFD because, due to hgh and fastchangng correlatons among VMs n our utlzaton traces, PCP classfes VMs nto only 1 cluster durng the most of the tme perods (22 out of 24 tme perods). When the number of clusters s 1, PCP behaves exactly same wth BFD. The power savngs obtaned by our proposed soluton are due to the aggressve-yet-safe v/f settngs utlzng the lowered actual peak resource demand,.e., Eqn. (4). Fg. 6 compares the dstrbutons of used frequency levels of BFD and the proposed soluton n two servers (we omt the dstrbuton of PCP, as t s smlar to BFD). As shown n the hstograms, the proposed soluton uses the lower frequency levels more frequently. Moreover, the proposed soluton provdes a drastc reducton of the volatons (.e., 15.6%) compared to the other approaches. Note that we allocated VMs based on ther peak utlzatons, whch were predcted from the ther hstory. Despte the provson based on the peak utlzaton, we observed qualty degradaton over the three approaches due to the ms-predctons of the peak utlzaton, especally durng abrupt workload changes. However, the proposed soluton can statstcally reduce the probablty of the volaton by co-locatng uncorrelated VMs. Thus, the probablty of jont under-predctons among the co-located VMs s drastcally decreased. To further nvestgate the effectveness of the proposed soluton, we also smulated the case of servers usng dynamc v/f scalng. To prevent frequent oscllatons of v/f level (whch affects server relablty [17]), we performed the v/f scalng at every 12 samples (.e., 1 mn). As shown n Table II(b), the power savngs become smaller compared to the statc v/f scalng because the other approaches also adaptvely scale v/f level accordng to the tme-varyng utlzaton demand. However, the amount of the volatons s unacceptably hgh n the other approaches. Thus, more servers need to be actvated to acheve the same QoS level obtaned by the proposed soluton, whch leads to hgher power consumpton. VI. CONCLUSIONS In ths paper, we have presented a novel dynamc power management soluton for datacenters targetng the executon of scale-out applcatons by jontly harnessng server consoldaton and v/f scalng, n order to reduce the global power consumpton whle satsfyng QoS requrements. Therefore, we have frst analyzed the characterstcs of scale-out applcatons and evaluated three fundamental approaches for datacenter dynamc power management solutons: 1) conservatve resource provson based on (off-)peak utlzaton, 2) sharng cores among co-located VMs, and 3) correlaton-aware VM placement. Then, we proposed a novel VM placement solutons utlzng the new defnton of correlaton and an aggressve-yet-safe v/f scalng soluton. Fnally, we valdated the applcablty of our proposed correlaton-aware scheme wth the applcaton of multple web search clusters n [10] and the utlzaton traces obtaned from real datacenter setups. Our expermental results show that the proposed soluton provdes up to 13.7% power savngs and up to 15.6% mprovement of QoS level compared to conventonal VM placement solutons. REFERENCES [1] R. H. Katz, Tech ttans buldng boom, n IEEE Spectrum, [2] A. Verma, et al., pmapper: power and mgraton cost aware applcaton placement n vrtualzed systems, n Proc. Mddleware [3] E. Pakbazna, et al., Mnmzng data center coolng and server power costs, n Proc. ISLPED, [4] N. Bobroff, et al., Dynamc placement of vrtual machnes for managng sla volatons, n Proc. IM [5] D. Mesner, et al., Power management of onlne data-ntensve servces, n Proc. ISCA, [6] A. Verma, et al., Server workload analyss for powr mnmzaton usng consoldaton, n Proc. USENIX, [7] X. Meng, et al., Effcent resource provsonng n compute clouds va VM multplexgn, n Proc. ICAC, [8] M. Chen, et al., Effectve VM szng n vrtualzed data centers, n Proc. IM, [9] K. Halder, et al., Rsk aware provsonng and resource aggregaton based consoldaton of vrtual machnes, n Proc. Cloud, [10] M. Ferdman, et al., Clearng the clouds: a study of emergng scale-out workloads on modern hardware, n Proc. ASPLOS, [11] E. Schurman et al., The user and busness mpact of server delays, addtonal bytes, and HTTP chunkng n web search, n Velocty, [12] H. Goudarz, et al., Energy-effcent vrtual machne replcaton and placement n a cloud computng system, n Proc. Cloud [13] M. Pedram, et al., Power and performance modelng n a vrtualzed server system, n Proc. ICPPW, [14] A. Menon, et al., Dganosng performance overheads n the xen vrtual machne envronment, n Proc. VEE, [15] J. Km, et al., Free coolng-aware dynamc power management for green datacenters, n Proc. HPCS, [16] T. Benson, et al., Understandng data center traffc characterstcs, n ACM SIGCOMM Computer Communcaton Revew, [17] Y. Guo, et al., Relablty-aware power management for parallel realtme applcatons wth precedence constrants, n Proc. IGCC, 2011.